diff --git a/docs/index.asciidoc b/docs/index.asciidoc index cddbf5e5699..2f32bb7e60d 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -187,14 +187,13 @@ include::static/deploying.asciidoc[] :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/performance-checklist.asciidoc include::static/performance-checklist.asciidoc[] -// Monitoring -:edit_url!: -include::static/monitoring/monitoring.asciidoc[] - // X-Pack Monitoring :edit_url!: include::static/monitoring/monitoring-overview.asciidoc[] +// Monitoring +:edit_url!: +include::static/monitoring/monitoring.asciidoc[] // Working with Plugins diff --git a/docs/static/docker.asciidoc b/docs/static/docker.asciidoc index a05f947a605..36690263245 100644 --- a/docs/static/docker.asciidoc +++ b/docs/static/docker.asciidoc @@ -153,7 +153,7 @@ Some example translations are shown here: **Environment Variable**:: **Logstash Setting** `PIPELINE_WORKERS`:: `pipeline.workers` `LOG_LEVEL`:: `log.level` -`XPACK_MONITORING_ENABLED`:: `xpack.monitoring.enabled` +`MONITORING_ENABLED`:: `monitoring.enabled` In general, any setting listed in the <> can be configured with this technique. @@ -170,9 +170,9 @@ images: [horizontal] `http.host`:: `0.0.0.0` -`xpack.monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200` +`monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200` -NOTE: The setting `xpack.monitoring.elasticsearch.hosts` is not +NOTE: The setting `monitoring.elasticsearch.hosts` is not defined in the `-oss` image. These settings are defined in the default `logstash.yml`. They can be overridden diff --git a/docs/static/monitoring/collectors-legacy.asciidoc b/docs/static/monitoring/collectors-legacy.asciidoc new file mode 100644 index 00000000000..1feb8baa4cc --- /dev/null +++ b/docs/static/monitoring/collectors-legacy.asciidoc @@ -0,0 +1,49 @@ +[float] +[role="xpack"] +[[logstash-monitoring-collectors-legacy]] +===== Collectors + +Collectors, as their name implies, collect things. In monitoring for Logstash, +collectors are just <> in the same way that ordinary Logstash +configurations provide inputs. + +Like monitoring for {es}, each collector can create zero or more monitoring +documents. As it is currently implemented, each Logstash node runs two types of +collectors: one for node stats and one for pipeline stats. + +[options="header"] +|======================= +| Collector | Data Types | Description +| Node Stats | `logstash_stats` +| Gathers details about the running node, such as memory utilization and CPU +usage (for example, `GET /_stats`). + +This runs on every Logstash node with monitoring enabled. One common +failure is that Logstash directories are copied with their `path.data` directory +included (`./data` by default), which copies the persistent UUID of the Logstash +node along with it. As a result, it generally appears that one or more Logstash +nodes are failing to collect monitoring data, when in fact they are all really +misreporting as the _same_ Logstash node. Re-use `path.data` directories only +when upgrading Logstash, such that upgraded nodes replace the previous versions. +| Pipeline Stats | `logstash_state` +| Gathers details about the node's running pipelines, which powers the +Monitoring Pipeline UI. +|======================= + +Per collection interval, which defaults to 10 seconds (`10s`), each collector is +run. The failure of an individual collector does not impact any other collector. +Each collector, as an ordinary Logstash input, creates a separate Logstash event +in its isolated monitoring pipeline. The Logstash output then sends the data. + +The collection interval can be configured dynamically and you can also disable +data collection. For more information about the configuration options for the +collectors, see <>. + +WARNING: Unlike {es} and {kib} monitoring, there is no +`xpack.monitoring.collection.enabled` setting on Logstash. You must use the +`xpack.monitoring.enabled` setting to enable and disable data collection. + +If gaps exist in the monitoring charts in {kib}, it is typically because either +a collector failed or the monitoring cluster did not receive the data (for +example, it was being restarted). In the event that a collector fails, a logged +error should exist on the node that attempted to perform the collection. diff --git a/docs/static/monitoring/collectors.asciidoc b/docs/static/monitoring/collectors.asciidoc index 30a373f24bb..8b09cf6a068 100644 --- a/docs/static/monitoring/collectors.asciidoc +++ b/docs/static/monitoring/collectors.asciidoc @@ -3,11 +3,11 @@ [[logstash-monitoring-collectors]] ===== Collectors -Collectors, as their name implies, collect things. In {monitoring} for Logstash, +Collectors, as their name implies, collect things. In monitoring for Logstash, collectors are just <> in the same way that ordinary Logstash configurations provide inputs. -Like {monitoring} for {es}, each collector can create zero or more monitoring +Like monitoring for {es}, each collector can create zero or more monitoring documents. As it is currently implemented, each Logstash node runs two types of collectors: one for node stats and one for pipeline stats. @@ -18,7 +18,7 @@ collectors: one for node stats and one for pipeline stats. | Gathers details about the running node, such as memory utilization and CPU usage (for example, `GET /_stats`). -This runs on every Logstash node with {monitoring} enabled. One common +This runs on every Logstash node with monitoring enabled. One common failure is that Logstash directories are copied with their `path.data` directory included (`./data` by default), which copies the persistent UUID of the Logstash node along with it. As a result, it generally appears that one or more Logstash @@ -39,9 +39,9 @@ The collection interval can be configured dynamically and you can also disable data collection. For more information about the configuration options for the collectors, see <>. -WARNING: Unlike {monitoring} for {es} and {kib}, there is no -`xpack.monitoring.collection.enabled` setting on Logstash. You must use the -`xpack.monitoring.enabled` setting to enable and disable data collection. +WARNING: Unlike for {es} and {kib} monitoring, there is no +`monitoring.collection.enabled` setting on Logstash. You must use the +`monitoring.enabled` setting to enable and disable data collection. If gaps exist in the monitoring charts in {kib}, it is typically because either a collector failed or the monitoring cluster did not receive the data (for diff --git a/docs/static/monitoring/monitoring-internal-legacy.asciidoc b/docs/static/monitoring/monitoring-internal-legacy.asciidoc new file mode 100644 index 00000000000..a86f8150170 --- /dev/null +++ b/docs/static/monitoring/monitoring-internal-legacy.asciidoc @@ -0,0 +1,155 @@ +[role="xpack"] +[[monitoring-internal-collection-legacy]] +=== Collect {ls} monitoring data using legacy internal collectors +++++ +Legacy internal collection +++++ + +IMPORTANT: Use <> or +<> instead of legacy +internal collection. If you are currently using legacy internal collection, you +should migrate to either Metricbeat collection or internal collection. + +==== Migrating from legacy internal collection to new internal collection + +Migrating from legacy internal collection to new +<> is straightforward. + +** Drop the `xpack.` prefix from your configuration settings. +For example, `xpack.monitoring.enabled` is now `monitoring.enabled`. See +<> for the full list. +** Change the hosts settings and configuration options to point to the monitoring +cluster instead of the production cluster. +** If you don’t have an Elasticsearch output plugin configured in the pipelines, +add the <> setting to your +logstash.yml. + +==== Components for legacy internal collection + +Monitoring {ls} with legacy internal collection uses these components: + +* <> +* <> + +These pieces live outside of the default Logstash pipeline in a dedicated monitoring +pipeline. This configuration ensures that all data and processing has a minimal +impact on ordinary Logstash processing. Existing Logstash features, such as the +<>, can be reused to +benefit from its retry policies. + +NOTE: The `elasticsearch` output that is used for monitoring {ls} is +configured exclusively through settings found in `logstash.yml`. It is not +configured by using anything from the Logstash configurations that might also be +using their own separate `elasticsearch` outputs. + + +The production {es} cluster should be configured to receive {ls} monitoring +data. This configuration enables the production {es} cluster to add metadata +(for example, its cluster UUID) to the Logstash monitoring data and then route +it to the monitoring clusters. For more information about typical monitoring +architectures, see {ref}/how-monitoring-works.html[How monitoring works] in the +{ref}[Elasticsearch Reference]. + + +include::collectors-legacy.asciidoc[] +include::monitoring-output-legacy.asciidoc[] + + +[[configure-internal-collectors-legacy]] +==== Configure {ls} monitoring with internal collectors +++++ +Configure internal collection +++++ + +To monitor Logstash nodes: + +. Specify where to send monitoring data. This cluster is often referred to as +the _production cluster_. For examples of typical monitoring architectures, see +{ref}/how-monitoring-works.html[How monitoring works]. ++ +-- +IMPORTANT: To visualize Logstash as part of the Elastic Stack (as shown in Step +6), send metrics to your _production_ cluster. Sending metrics to a dedicated +monitoring cluster will show the Logstash metrics under the _monitoring_ cluster. + +-- + +. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the +production cluster. If that setting is `false`, the collection of monitoring data +is disabled in {es} and data is ignored from all other sources. + +. Configure your Logstash nodes to send metrics by setting the +`xpack.monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features} +are enabled, you also need to specify the credentials for the +{ref}/built-in-users.html[built-in `logstash_system` user]. For more information +about these settings, see <>. ++ +-- +[source,yaml] +-------------------------------------------------- +xpack.monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1> +xpack.monitoring.elasticsearch.username: "logstash_system" +xpack.monitoring.elasticsearch.password: "changeme" +-------------------------------------------------- + +<1> If SSL/TLS is enabled on the production cluster, you must +connect through HTTPS. As of v5.2.1, you can specify multiple +Elasticsearch hosts as an array as well as specifying a single +host as a string. If multiple URLs are specified, Logstash +can round-robin requests to these production nodes. +-- + +. If SSL/TLS is enabled on the production {es} cluster, specify the trusted +CA certificates that will be used to verify the identity of the nodes +in the cluster. ++ +-- +To add a CA certificate to a Logstash node's trusted certificates, you +can specify the location of the PEM encoded certificate with the +`certificate_authority` setting: + +[source,yaml] +-------------------------------------------------- +xpack.monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt +-------------------------------------------------- + +Alternatively, you can configure trusted certificates using a truststore +(a Java Keystore file that contains the certificates): + +[source,yaml] +-------------------------------------------------- +xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file +xpack.monitoring.elasticsearch.ssl.truststore.password: password +-------------------------------------------------- + +Also, optionally, you can set up client certificate using a keystore +(a Java Keystore file that contains the certificate): + +[source,yaml] +-------------------------------------------------- +xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file +xpack.monitoring.elasticsearch.ssl.keystore.password: password +-------------------------------------------------- + +Set sniffing to `true` to enable discovery of other nodes of the {es} cluster. +It defaults to `false`. + +[source,yaml] +-------------------------------------------------- +xpack.monitoring.elasticsearch.sniffing: false +-------------------------------------------------- + +-- + +. Restart your Logstash nodes. + +. To verify your monitoring configuration, point your web browser at your {kib} +host, and select **Monitoring** from the side navigation. Metrics reported from +your Logstash nodes should be visible in the Logstash section. When security is +enabled, to view the monitoring dashboards you must log in to {kib} as a user +who has the `kibana_user` and `monitoring_user` roles. ++ +image::images/monitoring-ui.png["Monitoring",link="monitoring/images/monitoring-ui.png"] + +include::../settings/monitoring-settings-legacy.asciidoc[] + diff --git a/docs/static/monitoring/monitoring-internal.asciidoc b/docs/static/monitoring/monitoring-internal.asciidoc index f9eb3a8f87c..dd23d16bbc1 100644 --- a/docs/static/monitoring/monitoring-internal.asciidoc +++ b/docs/static/monitoring/monitoring-internal.asciidoc @@ -1,36 +1,15 @@ [role="xpack"] [[monitoring-internal-collection]] -=== Collect {ls} monitoring data using internal collectors +=== Use internal collectors to send monitoring data ++++ Internal collection ++++ -Using internal collectors for {ls} {monitoring} these components: - -* <> -* <> - -These pieces live outside of the default Logstash pipeline in a dedicated monitoring -pipeline. This configuration ensures that all data and processing has a minimal -impact on ordinary Logstash processing. Existing Logstash features, such as the -<>, can be reused to -benefit from its retry policies. - -NOTE: The `elasticsearch` output that is used by {monitoring} for Logstash is -configured exclusively via settings found in `logstash.yml`. It is not -configured by using anything from the Logstash configurations that might also be -using their own separate `elasticsearch` outputs. - -The {es} cluster that is configured for use with {monitoring} for Logstash is -expected to be the production cluster. This configuration enables the production -{es} cluster to add metadata (for example, its cluster UUID) to the Logstash -monitoring data then route it to the monitoring clusters. For more information -about typical monitoring architectures, see -{ref}/how-monitoring-works.html[How monitoring works] in the {ref}[Elasticsearch Reference]. - -include::collectors.asciidoc[] -include::monitoring-output.asciidoc[] +Internal collectors send {ls} monitoring data directly to your _monitoring_ cluster. +The benefit of internal collection is that you have fewer pieces of software to +install and maintain. +<> is available as an alternative. [[configure-internal-collectors]] ==== Configure {ls} monitoring with internal collectors @@ -40,23 +19,16 @@ include::monitoring-output.asciidoc[] To monitor Logstash nodes: -. Specify where to send monitoring data. This cluster is often referred to as -the _production cluster_. For examples of typical monitoring architectures, see -{ref}/how-monitoring-works.html[How monitoring works]. -+ --- -IMPORTANT: To visualize Logstash as part of the Elastic Stack (as shown in Step -6), send metrics to your _production_ cluster. Sending metrics to a dedicated -monitoring cluster will show the Logstash metrics under the _monitoring_ cluster. - --- +. Specify the location of the _monitoring cluster_. For examples of typical +monitoring architectures, see {ref}/how-monitoring-works.html[How monitoring +works] in the {ref}[Elasticsearch Reference]. -. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the -production cluster. If that setting is `false`, the collection of monitoring data -is disabled in {es} and data is ignored from all other sources. +. Verify that the `monitoring.collection.enabled` setting is `true` on the +monitoring cluster. If that setting is `false`, the collection of monitoring data +is disabled in {es}, and data is ignored from all other sources. . Configure your Logstash nodes to send metrics by setting the -`xpack.monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features} +`monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features} are enabled, you also need to specify the credentials for the {ref}/built-in-users.html[built-in `logstash_system` user]. For more information about these settings, see <>. @@ -64,21 +36,18 @@ information about these settings, see <>. -- [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1> -xpack.monitoring.elasticsearch.username: "logstash_system" <2> -xpack.monitoring.elasticsearch.password: "changeme" +monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] +monitoring.elasticsearch.username: "logstash_system" +monitoring.elasticsearch.password: "changeme" -------------------------------------------------- -<1> If SSL/TLS is enabled on the production cluster, you must -connect through HTTPS. As of v5.2.1, you can specify multiple -Elasticsearch hosts as an array as well as specifying a single -host as a string. If multiple URLs are specified, Logstash -can round-robin requests to these production nodes. -<2> If {security-features} are disabled on the production cluster, you can omit -these `username` and `password` settings. +If SSL/TLS is enabled on the monitoring cluster, you must connect through HTTPS. +You can specify a single host as a string, or multiple Elasticsearch hosts as an +array. If multiple URLs are specified, Logstash can round-robin requests to +these monitoring nodes. -- -. If SSL/TLS is enabled on the production {es} cluster, specify the trusted +. If SSL/TLS is enabled on the monitoring {es} cluster, specify the trusted CA certificates that will be used to verify the identity of the nodes in the cluster. + @@ -89,7 +58,7 @@ can specify the location of the PEM encoded certificate with the [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt +monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt -------------------------------------------------- Alternatively, you can configure trusted certificates using a truststore @@ -97,8 +66,8 @@ Alternatively, you can configure trusted certificates using a truststore [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file -xpack.monitoring.elasticsearch.ssl.truststore.password: password +monitoring.elasticsearch.ssl.truststore.path: /path/to/file +monitoring.elasticsearch.ssl.truststore.password: password -------------------------------------------------- Also, optionally, you can set up client certificate using a keystore @@ -106,8 +75,8 @@ Also, optionally, you can set up client certificate using a keystore [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file -xpack.monitoring.elasticsearch.ssl.keystore.password: password +monitoring.elasticsearch.ssl.keystore.path: /path/to/file +monitoring.elasticsearch.ssl.keystore.password: password -------------------------------------------------- Set sniffing to `true` to enable discovery of other nodes of the {es} cluster. @@ -115,28 +84,46 @@ It defaults to `false`. [source,yaml] -------------------------------------------------- -xpack.monitoring.elasticsearch.sniffing: false +monitoring.elasticsearch.sniffing: false -------------------------------------------------- -- . Restart your Logstash nodes. -. To verify your {monitoring} configuration, point your web browser at your {kib} +. To verify your monitoring configuration, point your web browser at your {kib} host, and select **Monitoring** from the side navigation. Metrics reported from your Logstash nodes should be visible in the Logstash section. When security is -enabled, to view the monitoring dashboards you must log in to {kib} as a user -who has the `kibana_user` and `monitoring_user` roles. +enabled, you must log in to {kib} as a user who has the `kibana_user` and +`monitoring_user` roles. + image::images/monitoring-ui.png["Monitoring",link="monitoring/images/monitoring-ui.png"] -[float] -[[monitoring-upgraded-logstash]] -===== Re-enabling Logstash Monitoring After Upgrading +include::../settings/monitoring-settings.asciidoc[] -When upgrading from older versions of {xpack}, the built-in `logstash_system` -user is disabled for security reasons. To resume monitoring, -change the password and re-enable the logstash_system user. -include::../settings/monitoring-settings.asciidoc[] +[[internal-collector-components]] +==== How {ls} monitoring with internal collectors works + +Monitoring {ls} with internal collectors uses these components: + +* <> +* <> +These pieces live outside of the default Logstash pipeline in a dedicated +monitoring pipeline. This configuration ensures that all data and processing has +a minimal impact on ordinary Logstash processing. Existing Logstash features, +such as the <>, can be +reused to benefit from its retry policies. + +NOTE: The `elasticsearch` output for Logstash monitoring is configured +exclusively through settings in `logstash.yml`. + +The monitoring {es} cluster should be configured to receive {ls} monitoring +data directly from {ls}. For more information about typical monitoring +architectures, see {ref}/how-monitoring-works.html[How monitoring works] in the +{ref}[Elasticsearch Reference]. + + +include::collectors.asciidoc[] +include::monitoring-output.asciidoc[] diff --git a/docs/static/monitoring/monitoring-mb.asciidoc b/docs/static/monitoring/monitoring-mb.asciidoc index ba54c04ada3..dec7f692590 100644 --- a/docs/static/monitoring/monitoring-mb.asciidoc +++ b/docs/static/monitoring/monitoring-mb.asciidoc @@ -6,16 +6,19 @@ {metricbeat} collection ++++ -You can use {metricbeat} to collect data about {ls} -and ship it to the monitoring cluster, rather than routing it through the -production cluster as described in <>. +You can use {metricbeat} to collect data about {ls} and ship it to the +monitoring cluster. The benefit of Metricbeat collection is that the monitoring +agent remains active even if the {ls} instance does not. + +<> is available as an +alternative. //NOTE: The tagged regions are re-used in the Stack Overview. To collect and ship monitoring data: . <> -. <> +. <> . <> [float] @@ -29,7 +32,7 @@ commented out: [source,yaml] ---------------------------------- -xpack.monitoring.enabled: false +monitoring.enabled: false ---------------------------------- Remove the `#` at the beginning of the line to enable the setting. diff --git a/docs/static/monitoring/monitoring-output-legacy.asciidoc b/docs/static/monitoring/monitoring-output-legacy.asciidoc new file mode 100644 index 00000000000..349308f62ed --- /dev/null +++ b/docs/static/monitoring/monitoring-output-legacy.asciidoc @@ -0,0 +1,47 @@ +[float] +[role="xpack"] +[[logstash-monitoring-output-legacy]] +==== Output + +Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline is +to send events to outputs. In the case of monitoring for Logstash, the output +is always an `elasticsearch` output. However, unlike ordinary Logstash pipelines, +the output is configured within the `logstash.yml` settings file via the +`xpack.monitoring.elasticsearch.*` settings. + +Other than its unique manner of configuration, this `elasticsearch` output +behaves like all `elasticsearch` outputs, including its ability to pause data +collection when issues exist with the output. + +IMPORTANT: It is critical that all Logstash nodes share the same setup. +Otherwise, monitoring data might be routed in different ways or to different places. + +[float] +[[logstash-monitoring-default-legacy]] +===== Default Configuration + +If a Logstash node does not explicitly define a monitoring output setting, +the following default configuration is used: + +[source,yaml] +--------------------------------------------------- +xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ] +--------------------------------------------------- + +All data produced by monitoring for Logstash is indexed in the monitoring +cluster by using the `.monitoring-logstash` template, which is managed by the +{ref}/es-monitoring-exporters.html[exporters] within {es}. + +If you are working with a cluster that has {security} enabled, extra steps are +necessary to properly configure Logstash. For more information, see +<>. + +IMPORTANT: When discussing security relative to the `elasticsearch` output, it +is critical to remember that all users are managed on the production cluster, +which is identified in the `xpack.monitoring.elasticsearch.hosts` setting. +This is particularly important to remember when you move from development +environments to production environments, where you often have dedicated +monitoring clusters. + +For more information about the configuration options for the output, see +<>. \ No newline at end of file diff --git a/docs/static/monitoring/monitoring-output.asciidoc b/docs/static/monitoring/monitoring-output.asciidoc index 16ba44ec9a6..18a226351b3 100644 --- a/docs/static/monitoring/monitoring-output.asciidoc +++ b/docs/static/monitoring/monitoring-output.asciidoc @@ -4,10 +4,10 @@ ==== Output Like all Logstash pipelines, the purpose of the dedicated monitoring pipeline is -to send events to outputs. In the case of {monitoring} for Logstash, the output +to send events to outputs. In the case of Logstash monitoring, the output is always an `elasticsearch` output. However, unlike ordinary Logstash pipelines, the output is configured within the `logstash.yml` settings file via the -`xpack.monitoring.elasticsearch.*` settings. +`monitoring.elasticsearch.*` settings. Other than its unique manner of configuration, this `elasticsearch` output behaves like all `elasticsearch` outputs, including its ability to pause data @@ -20,15 +20,15 @@ Otherwise, monitoring data might be routed in different ways or to different pla [[logstash-monitoring-default]] ===== Default Configuration -If a Logstash node does not explicitly define an {monitoring} output setting, +If a Logstash node does not explicitly define a monitoring output setting, the following default configuration is used: [source,yaml] --------------------------------------------------- -xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ] +monitoring.elasticsearch.hosts: [ "http://localhost:9200" ] --------------------------------------------------- -All data produced by {monitoring} for Logstash is indexed in the monitoring +All data produced by Logstash monitoring is indexed in the monitoring cluster by using the `.monitoring-logstash` template, which is managed by the {ref}/es-monitoring-exporters.html[exporters] within {es}. @@ -36,12 +36,11 @@ If you are working with a cluster that has {security-features} enabled, extra steps are necessary to properly configure Logstash. For more information, see <>. -IMPORTANT: When discussing security relative to the `elasticsearch` output, it -is critical to remember that all users are managed on the production cluster, -which is identified in the `xpack.monitoring.elasticsearch.hosts` setting. -This is particularly important to remember when you move from development -environments to production environments, where you often have dedicated -monitoring clusters. +IMPORTANT: When discussing security relative to the `elasticsearch` output, +remember that all users are managed on the production cluster, which is +identified in the `monitoring.elasticsearch.hosts` setting. This is particularly +important when you move from development environments to production +environments, where you often have dedicated monitoring clusters. For more information about the configuration options for the output, see <>. \ No newline at end of file diff --git a/docs/static/monitoring/monitoring-overview.asciidoc b/docs/static/monitoring/monitoring-overview.asciidoc index 1c6ff5b9bdf..5e6e994ee52 100644 --- a/docs/static/monitoring/monitoring-overview.asciidoc +++ b/docs/static/monitoring/monitoring-overview.asciidoc @@ -1,27 +1,40 @@ [role="xpack"] [[configuring-logstash]] -== Monitoring {ls} with {xpack} +== Monitoring {ls} Use the {stack} {monitor-features} to gain insight into the health of {ls} instances running in your environment. For an introduction to monitoring your Elastic stack, see -{ref}/monitor-elasticsearch-cluster.html[Monitoring a cluster]. - +{ref}/monitor-elasticsearch-cluster.html[Monitoring a cluster] in the +{ref}[Elasticsearch Reference]. [float] [[configuring-logstash-xpack]] -=== Configuring {xpack} monitoring for {ls} +=== Configuring monitoring for {ls} Make sure monitoring is enabled on your {es} cluster. Then configure *one* of these methods to collect {ls} metrics: -* <> -* <> +* <>. Metricbeat collects +monitoring data from your {ls} instance and sends it directly to your monitoring +cluster. The benefit of Metricbeat collection is that the monitoring +agent remains active even if the {ls} instance does not. + +* <>. Internal collectors send +monitoring data directly to your monitoring cluster. The benefit of internal +collection is that you have fewer pieces of software to install and maintain. + +* <>. Legacy +internal collectors send monitoring data to your production cluster. +This is no longer a preferred approach. Use +<> or +<> instead. include::monitoring-mb.asciidoc[] include::monitoring-internal.asciidoc[] +include::monitoring-internal-legacy.asciidoc[] include::monitoring-ui.asciidoc[] include::pipeline-viewer.asciidoc[] include::troubleshooting.asciidoc[] diff --git a/docs/static/monitoring/monitoring.asciidoc b/docs/static/monitoring/monitoring.asciidoc index 6b85a78304c..11c1a4c669f 100644 --- a/docs/static/monitoring/monitoring.asciidoc +++ b/docs/static/monitoring/monitoring.asciidoc @@ -19,7 +19,7 @@ requiring any extra configuration. Alternatively, you can <> to send data to a monitoring cluster. -NOTE: {monitoring} is a feature under the Basic License and is therefore +NOTE: Monitoring is a feature under the Basic License and is therefore *free to use*. diff --git a/docs/static/monitoring/troubleshooting.asciidoc b/docs/static/monitoring/troubleshooting.asciidoc index 19099d34bef..40cf01a2522 100644 --- a/docs/static/monitoring/troubleshooting.asciidoc +++ b/docs/static/monitoring/troubleshooting.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [[monitoring-troubleshooting]] -=== Troubleshooting {monitoring} in Logstash +=== Troubleshooting monitoring in Logstash ++++ Troubleshooting ++++ diff --git a/docs/static/security/logstash.asciidoc b/docs/static/security/logstash.asciidoc index 0328d6a6f84..fa19d62f1a9 100644 --- a/docs/static/security/logstash.asciidoc +++ b/docs/static/security/logstash.asciidoc @@ -217,8 +217,8 @@ Then configure the user and password in the `logstash.yml` configuration file: [source,yaml] ---------------------------------------------------------- -xpack.monitoring.elasticsearch.username: logstash_system -xpack.monitoring.elasticsearch.password: t0p.s3cr3t +monitoring.elasticsearch.username: logstash_system +monitoring.elasticsearch.password: t0p.s3cr3t ---------------------------------------------------------- If you initially installed an older version of {xpack} and then upgraded, the diff --git a/docs/static/settings/monitoring-settings-legacy.asciidoc b/docs/static/settings/monitoring-settings-legacy.asciidoc new file mode 100644 index 00000000000..686c237bd12 --- /dev/null +++ b/docs/static/settings/monitoring-settings-legacy.asciidoc @@ -0,0 +1,106 @@ +[role="xpack"] +[[monitoring-settings-legacy]] +==== Monitoring settings for legacy internal collection +++++ +Monitoring Settings +++++ + +You can set the following `xpack.monitoring` settings in `logstash.yml` to +control how monitoring data is collected from your Logstash nodes. However, the +defaults work best in most circumstances. For more information about configuring +Logstash, see <>. + + +[[monitoring-general-settings-legacy]] +===== General monitoring settings + +`xpack.monitoring.enabled`:: + +Monitoring is disabled by default. Set to `true` to enable {xpack} monitoring. + +`xpack.monitoring.elasticsearch.hosts`:: + +The {es} instances that you want to ship your Logstash metrics to. This might be +the same {es} instance specified in the `outputs` section in your Logstash +configuration, or a different one. This is *not* the URL of your dedicated +monitoring cluster. Even if you are using a dedicated monitoring cluster, the +Logstash metrics must be routed through your production cluster. You can specify +a single host as a string, or specify multiple hosts as an array. Defaults to +`http://localhost:9200`. + +NOTE: If your Elasticsearch cluster is configured with dedicated master-eliglble +nodes, Logstash metrics should _not_ be routed to these nodes, as doing so can +create resource contention and impact the stability of the Elasticsearch +cluster. Therefore, do not include such nodes in +`xpack.monitoring.elasticsearch.hosts`. + +`xpack.monitoring.elasticsearch.username` and `xpack.monitoring.elasticsearch.password`:: + +If your {es} is protected with basic authentication, these settings provide the +username and password that the Logstash instance uses to authenticate for +shipping monitoring data. + + +[[monitoring-collection-settings-legacy]] +===== Monitoring collection settings + +`xpack.monitoring.collection.interval`:: + +Controls how often data samples are collected and shipped on the Logstash side. +Defaults to `10s`. If you modify the collection interval, set the +`xpack.monitoring.min_interval_seconds` option in `kibana.yml` to the same value. + + +[[monitoring-ssl-settings-legacy]] +===== Monitoring TLS/SSL settings + +You can configure the following Transport Layer Security (TLS) or +Secure Sockets Layer (SSL) settings. For more information, see +<>. + +`xpack.monitoring.elasticsearch.ssl.certificate_authority`:: + +Optional setting that enables you to specify a path to the `.pem` file for the +certificate authority for your {es} instance. + +`xpack.monitoring.elasticsearch.ssl.truststore.path`:: + +Optional settings that provide the paths to the Java keystore (JKS) to validate +the server’s certificate. + +`xpack.monitoring.elasticsearch.ssl.truststore.password`:: + +Optional settings that provide the password to the truststore. + +`xpack.monitoring.elasticsearch.ssl.keystore.path`:: + +Optional settings that provide the paths to the Java keystore (JKS) to validate +the client’s certificate. + +`xpack.monitoring.elasticsearch.ssl.keystore.password`:: + +Optional settings that provide the password to the keystore. + +`xpack.monitoring.elasticsearch.ssl.verification_mode`:: + +Option to validate the server’s certificate. Defaults to `certificate`. To +disable, set to `none`. Disabling this severely compromises security. + +[[monitoring-additional-settings-legacy]] +===== Additional settings + +`xpack.monitoring.elasticsearch.cloud_id`:: + +If you're using {es} in {ecloud}, you should specify the identifier here. +This setting is an alternative to `xpack.monitoring.elasticsearch.hosts`. +If `cloud_id` is configured, `xpack.monitoring.elasticsearch.hosts` should not be used. +The {es} instances that you want to ship your Logstash metrics to. This might be +the same {es} instance specified in the `outputs` section in your Logstash +configuration, or a different one. + +`xpack.monitoring.elasticsearch.cloud_auth`:: + +If you're using {es} in {ecloud}, you can set your auth credentials here. +This setting is an alternative to both `xpack.monitoring.elasticsearch.username` +and `xpack.monitoring.elasticsearch.password`. If `cloud_auth` is configured, +those settings should not be used. diff --git a/docs/static/settings/monitoring-settings.asciidoc b/docs/static/settings/monitoring-settings.asciidoc index 0574663f9f1..26caee9bbe0 100644 --- a/docs/static/settings/monitoring-settings.asciidoc +++ b/docs/static/settings/monitoring-settings.asciidoc @@ -5,7 +5,7 @@ Monitoring Settings ++++ -You can set the following `xpack.monitoring` settings in `logstash.yml` to +You can set the following `monitoring` settings in `logstash.yml` to control how monitoring data is collected from your Logstash nodes. However, the defaults work best in most circumstances. For more information about configuring Logstash, see <>. @@ -14,27 +14,25 @@ Logstash, see <>. [[monitoring-general-settings]] ===== General monitoring settings -`xpack.monitoring.enabled`:: +`monitoring.enabled`:: -Monitoring is disabled by default. Set to `true` to enable {xpack} monitoring. +Monitoring is disabled by default. Set to `true` to enable monitoring. -`xpack.monitoring.elasticsearch.hosts`:: +`monitoring.elasticsearch.hosts`:: -The {es} instances that you want to ship your Logstash metrics to. This might be -the same {es} instance specified in the `outputs` section in your Logstash -configuration, or a different one. This is *not* the URL of your dedicated -monitoring cluster. Even if you are using a dedicated monitoring cluster, the -Logstash metrics must be routed through your production cluster. You can specify -a single host as a string, or specify multiple hosts as an array. Defaults to +The {es} monitoring cluster that you want to ship your Logstash metrics to. This +might be the same {es} instance specified in the `outputs` section in your +Logstash configuration, or a dedicated monitoring cluster. You can specify a +single host as a string, or specify multiple hosts as an array. Defaults to `http://localhost:9200`. -NOTE: If your Elasticsearch cluster is configured with dedicated master-eliglble +NOTE: If your Elasticsearch cluster is configured with dedicated master-eligible nodes, Logstash metrics should _not_ be routed to these nodes, as doing so can create resource contention and impact the stability of the Elasticsearch cluster. Therefore, do not include such nodes in -`xpack.monitoring.elasticsearch.hosts`. +`monitoring.elasticsearch.hosts`. -`xpack.monitoring.elasticsearch.username` and `xpack.monitoring.elasticsearch.password`:: +`monitoring.elasticsearch.username` and `monitoring.elasticsearch.password`:: If your {es} is protected with basic authentication, these settings provide the username and password that the Logstash instance uses to authenticate for @@ -44,65 +42,80 @@ shipping monitoring data. [[monitoring-collection-settings]] ===== Monitoring collection settings -`xpack.monitoring.collection.interval`:: +`monitoring.collection.interval`:: Controls how often data samples are collected and shipped on the Logstash side. Defaults to `10s`. If you modify the collection interval, set the -`xpack.monitoring.min_interval_seconds` option in `kibana.yml` to the same value. +`monitoring.min_interval_seconds` option in `kibana.yml` to the same value. + +[[monitoring-cluster-uuid]] +`monitoring.cluster_uuid`:: +The universally unique identifier (UUID) for the monitoring cluster. +By default, {ls} identifies and uses the `cluster uuid` value from each +elasticsearch output defined in the pipelines, and ignores this +setting. ++ +If no `cluster_uuid` is discovered in elasticsearch outputs, then {ls} +uses this value to tag the data shipped to the monitoring cluster. [[monitoring-ssl-settings]] -===== {monitoring} TLS/SSL settings +===== Monitoring TLS/SSL settings You can configure the following Transport Layer Security (TLS) or Secure Sockets Layer (SSL) settings. For more information, see <>. -`xpack.monitoring.elasticsearch.ssl.certificate_authority`:: +`monitoring.elasticsearch.ssl.certificate_authority`:: Optional setting that enables you to specify a path to the `.pem` file for the certificate authority for your {es} instance. -`xpack.monitoring.elasticsearch.ssl.truststore.path`:: +`monitoring.elasticsearch.ssl.truststore.path`:: Optional settings that provide the paths to the Java keystore (JKS) to validate the server’s certificate. -`xpack.monitoring.elasticsearch.ssl.truststore.password`:: +`monitoring.elasticsearch.ssl.truststore.password`:: Optional settings that provide the password to the truststore. -`xpack.monitoring.elasticsearch.ssl.keystore.path`:: +`monitoring.elasticsearch.ssl.keystore.path`:: Optional settings that provide the paths to the Java keystore (JKS) to validate the client’s certificate. -`xpack.monitoring.elasticsearch.ssl.keystore.password`:: +`monitoring.elasticsearch.ssl.keystore.password`:: Optional settings that provide the password to the keystore. -`xpack.monitoring.elasticsearch.ssl.verification_mode`:: +`monitoring.elasticsearch.ssl.verification_mode`:: Option to validate the server’s certificate. Defaults to `certificate`. To disable, set to `none`. Disabling this severely compromises security. +`monitoring.elasticsearch.sniffing`:: + +Finds all {es} cluster nodes and adds them to the hosts list. +Defaults to `false`. + [[monitoring-additional-settings]] ===== Additional settings -`xpack.monitoring.elasticsearch.cloud_id`:: +`monitoring.elasticsearch.cloud_id`:: If you're using {es} in {ecloud}, you should specify the identifier here. -This setting is an alternative to `xpack.monitoring.elasticsearch.hosts`. -If `cloud_id` is configured, `xpack.monitoring.elasticsearch.hosts` should not be used. +This setting is an alternative to `monitoring.elasticsearch.hosts`. +If `cloud_id` is configured, `monitoring.elasticsearch.hosts` should not be used. The {es} instances that you want to ship your Logstash metrics to. This might be the same {es} instance specified in the `outputs` section in your Logstash configuration, or a different one. -`xpack.monitoring.elasticsearch.cloud_auth`:: +`monitoring.elasticsearch.cloud_auth`:: If you're using {es} in {ecloud}, you can set your auth credentials here. -This setting is an alternative to both `xpack.monitoring.elasticsearch.username` -and `xpack.monitoring.elasticsearch.password`. If `cloud_auth` is configured, +This setting is an alternative to both `monitoring.elasticsearch.username` +and `monitoring.elasticsearch.password`. If `cloud_auth` is configured, those settings should not be used. diff --git a/docs/static/upgrading.asciidoc b/docs/static/upgrading.asciidoc index c81e4b5a9eb..9583013897f 100644 --- a/docs/static/upgrading.asciidoc +++ b/docs/static/upgrading.asciidoc @@ -11,7 +11,7 @@ Before upgrading Logstash: While upgrading Logstash: -* If you use {monitoring}, you must re-use the data directory when you +* If you use monitoring, you must re-use the data directory when you upgrade Logstash. Otherwise, the Logstash node is assigned a new persistent UUID and becomes a new node in the monitoring data. ===========================================