[Doc]Restructure monitoring docs to support new and legacy internal collectors

docs/index.asciidoc

@@ -187,14 +187,13 @@ include::static/deploying.asciidoc[]
 :edit_url: https://github.com/elastic/logstash/edit/{branch}/docs/static/performance-checklist.asciidoc
 include::static/performance-checklist.asciidoc[]
 
-// Monitoring 
-:edit_url!:
-include::static/monitoring/monitoring.asciidoc[]
-
 // X-Pack Monitoring 
 :edit_url!:
 include::static/monitoring/monitoring-overview.asciidoc[]
 
+// Monitoring 
+:edit_url!:
+include::static/monitoring/monitoring.asciidoc[]
 
 // Working with Plugins
 

docs/static/docker.asciidoc

@@ -153,7 +153,7 @@ Some example translations are shown here:
 **Environment Variable**:: **Logstash Setting**
 `PIPELINE_WORKERS`:: `pipeline.workers`
 `LOG_LEVEL`:: `log.level`
-`XPACK_MONITORING_ENABLED`:: `xpack.monitoring.enabled`
+`MONITORING_ENABLED`:: `monitoring.enabled`
 
 In general, any setting listed in the <<logstash-settings-file, settings
 documentation>> can be configured with this technique.
@@ -170,9 +170,9 @@ images:
 
 [horizontal]
 `http.host`:: `0.0.0.0`
-`xpack.monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200`
+`monitoring.elasticsearch.hosts`:: `http://elasticsearch:9200`
 
-NOTE: The setting `xpack.monitoring.elasticsearch.hosts` is not
+NOTE: The setting `monitoring.elasticsearch.hosts` is not
 defined in the `-oss` image.
 
 These settings are defined in the default `logstash.yml`. They can be overridden

docs/static/monitoring/collectors-legacy.asciidoc

@@ -0,0 +1,49 @@
+[float]
+[role="xpack"]
+[[logstash-monitoring-collectors-legacy]]
+===== Collectors
+
+Collectors, as their name implies, collect things. In monitoring for Logstash, 
+collectors are just <<pipeline,Inputs>> in the same way that ordinary Logstash 
+configurations provide inputs.
+
+Like monitoring for {es}, each collector can create zero or more monitoring 
+documents. As it is currently implemented, each Logstash node runs two types of 
+collectors: one for node stats and one for pipeline stats.
+
+[options="header"]
+|=======================
+| Collector       | Data Types | Description
+| Node Stats      | `logstash_stats`
+| Gathers details about the running node, such as memory utilization and CPU
+usage (for example, `GET /_stats`).
+
+This runs on every Logstash node with monitoring enabled. One common
+failure is that Logstash directories are copied with their `path.data` directory
+included (`./data` by default), which copies the persistent UUID of the Logstash
+node along with it. As a result, it generally appears that one or more Logstash
+nodes are failing to collect monitoring data, when in fact they are all really
+misreporting as the _same_ Logstash node. Re-use `path.data` directories only 
+when upgrading Logstash, such that upgraded nodes replace the previous versions.
+| Pipeline Stats | `logstash_state`
+| Gathers details about the node's running pipelines, which powers the 
+Monitoring Pipeline UI.
+|=======================
+
+Per collection interval, which defaults to 10 seconds (`10s`), each collector is
+run. The failure of an individual collector does not impact any other collector. 
+Each collector, as an ordinary Logstash input, creates a separate Logstash event 
+in its isolated monitoring pipeline. The Logstash output then sends the data.
+
+The collection interval can be configured dynamically and you can also disable 
+data collection. For more information about the configuration options for the 
+collectors, see <<monitoring-settings-legacy>>.
+
+WARNING: Unlike {es} and {kib} monitoring, there is no 
+`xpack.monitoring.collection.enabled` setting on Logstash. You must use the 
+`xpack.monitoring.enabled` setting to enable and disable data collection. 
+
+If gaps exist in the monitoring charts in {kib}, it is typically because either 
+a collector failed or the monitoring cluster did not receive the data (for 
+example, it was being restarted). In the event that a collector fails, a logged 
+error should exist on the node that attempted to perform the collection.

docs/static/monitoring/collectors.asciidoc

@@ -3,11 +3,11 @@
 [[logstash-monitoring-collectors]]
 ===== Collectors
 
-Collectors, as their name implies, collect things. In {monitoring} for Logstash, 
+Collectors, as their name implies, collect things. In monitoring for Logstash, 
 collectors are just <<pipeline,Inputs>> in the same way that ordinary Logstash 
 configurations provide inputs.
 
-Like {monitoring} for {es}, each collector can create zero or more monitoring 
+Like monitoring for {es}, each collector can create zero or more monitoring 
 documents. As it is currently implemented, each Logstash node runs two types of 
 collectors: one for node stats and one for pipeline stats.
 
@@ -18,7 +18,7 @@ collectors: one for node stats and one for pipeline stats.
 | Gathers details about the running node, such as memory utilization and CPU
 usage (for example, `GET /_stats`).
 
-This runs on every Logstash node with {monitoring} enabled. One common
+This runs on every Logstash node with monitoring enabled. One common
 failure is that Logstash directories are copied with their `path.data` directory
 included (`./data` by default), which copies the persistent UUID of the Logstash
 node along with it. As a result, it generally appears that one or more Logstash
@@ -39,9 +39,9 @@ The collection interval can be configured dynamically and you can also disable
 data collection. For more information about the configuration options for the 
 collectors, see <<monitoring-settings>>.
 
-WARNING: Unlike {monitoring} for {es} and {kib}, there is no 
-`xpack.monitoring.collection.enabled` setting on Logstash. You must use the 
-`xpack.monitoring.enabled` setting to enable and disable data collection. 
+WARNING: Unlike for {es} and {kib} monitoring, there is no
+`monitoring.collection.enabled` setting on Logstash. You must use the
+`monitoring.enabled` setting to enable and disable data collection.
 
 If gaps exist in the monitoring charts in {kib}, it is typically because either 
 a collector failed or the monitoring cluster did not receive the data (for 

docs/static/monitoring/monitoring-internal-legacy.asciidoc

@@ -0,0 +1,155 @@
+[role="xpack"]
+[[monitoring-internal-collection-legacy]]
+=== Collect {ls} monitoring data using legacy internal collectors
+++++
+<titleabbrev>Legacy internal collection</titleabbrev>
+++++
+
+IMPORTANT: Use <<monitoring-internal-collection,internal collection>> or
+<<monitoring-with-metricbeat, {metricbeat} collection>> instead of legacy
+internal collection. If you are currently using legacy internal collection, you
+should migrate to either Metricbeat collection or internal collection.
+
+==== Migrating from legacy internal collection to new internal collection
+
+Migrating from legacy internal collection to new
+<<monitoring-internal-collection,Internal collection>> is straightforward. 
+
+** Drop the `xpack.` prefix from your configuration settings. 
+For example, `xpack.monitoring.enabled` is now `monitoring.enabled`. See
+<<monitoring-settings, Monitoring Settings>> for the full list.
+** Change the hosts settings and configuration options to point to the monitoring
+cluster instead of the production cluster.
+** If you don’t have an Elasticsearch output plugin configured in the pipelines,
+add the <<monitoring-cluster-uuid,`monitoring.cluster_uuid`>> setting to your 
+logstash.yml.
+
+==== Components for legacy internal collection
+
+Monitoring {ls} with legacy internal collection uses these components:
+
+* <<logstash-monitoring-collectors-legacy,Collectors>>
+* <<logstash-monitoring-output-legacy,Output>>
+
+These pieces live outside of the default Logstash pipeline in a dedicated monitoring
+pipeline. This configuration ensures that all data and processing has a minimal
+impact on ordinary Logstash processing. Existing Logstash features, such as the
+<<plugins-outputs-elasticsearch,`elasticsearch` output>>, can be reused to
+benefit from its retry policies. 
+
+NOTE: The `elasticsearch` output that is used for monitoring {ls} is 
+configured exclusively through settings found in `logstash.yml`. It is not 
+configured by using anything from the Logstash configurations that might also be 
+using their own separate `elasticsearch` outputs.
+
+
+The production {es} cluster should be configured to receive {ls} monitoring
+data. This configuration enables the production {es} cluster to add metadata
+(for example, its cluster UUID) to the Logstash monitoring data and then route
+it to the monitoring clusters. For more information  about typical monitoring
+architectures, see  {ref}/how-monitoring-works.html[How monitoring works] in the
+{ref}[Elasticsearch Reference]. 
+
+
+include::collectors-legacy.asciidoc[]
+include::monitoring-output-legacy.asciidoc[]
+
+
+[[configure-internal-collectors-legacy]]
+==== Configure {ls} monitoring with internal collectors
+++++
+<titleabbrev>Configure internal collection</titleabbrev>
+++++
+
+To monitor Logstash nodes:
+
+. Specify where to send monitoring data. This cluster is often referred to as
+the _production cluster_. For examples of typical monitoring architectures, see
+{ref}/how-monitoring-works.html[How monitoring works].
++
+--
+IMPORTANT: To visualize Logstash as part of the Elastic Stack (as shown in Step
+6), send metrics to your _production_ cluster. Sending metrics to a dedicated
+monitoring cluster will show the Logstash metrics under the _monitoring_ cluster.
+
+--
+
+. Verify that the `xpack.monitoring.collection.enabled` setting is `true` on the
+production cluster. If that setting is `false`, the collection of monitoring data
+is disabled in {es} and data is ignored from all other sources.
+
+. Configure your Logstash nodes to send metrics by setting the
+`xpack.monitoring.elasticsearch.hosts` in `logstash.yml`. If {security-features}
+are enabled, you also need to specify the credentials for the
+{ref}/built-in-users.html[built-in `logstash_system` user]. For more information
+about these settings, see <<monitoring-settings-legacy>>.
++
+--
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.elasticsearch.hosts: ["http://es-prod-node-1:9200", "http://es-prod-node-2:9200"] <1>
+xpack.monitoring.elasticsearch.username: "logstash_system"
+xpack.monitoring.elasticsearch.password: "changeme"
+--------------------------------------------------
+
+<1> If SSL/TLS is enabled on the production cluster, you must
+connect through HTTPS. As of v5.2.1, you can specify multiple
+Elasticsearch hosts as an array as well as specifying a single
+host as a string. If multiple URLs are specified, Logstash
+can round-robin requests to these production nodes. 
+--
+
+. If SSL/TLS is enabled on the production {es} cluster, specify the trusted
+CA certificates that will be used to verify the identity of the nodes
+in the cluster.
++
+--
+To add a CA certificate to a Logstash node's trusted certificates, you
+can specify the location of the PEM encoded certificate with the
+`certificate_authority` setting:
+
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.elasticsearch.ssl.certificate_authority: /path/to/ca.crt
+--------------------------------------------------
+
+Alternatively, you can configure trusted certificates using a truststore
+(a Java Keystore file that contains the certificates):
+
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.elasticsearch.ssl.truststore.path: /path/to/file
+xpack.monitoring.elasticsearch.ssl.truststore.password: password
+--------------------------------------------------
+
+Also, optionally, you can set up client certificate using a keystore
+(a Java Keystore file that contains the certificate):
+
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
+xpack.monitoring.elasticsearch.ssl.keystore.password: password
+--------------------------------------------------
+
+Set sniffing to `true` to enable discovery of other nodes of the {es} cluster.
+It defaults to `false`.
+
+[source,yaml]
+--------------------------------------------------
+xpack.monitoring.elasticsearch.sniffing: false
+--------------------------------------------------
+
+--
+
+. Restart your Logstash nodes.
+
+. To verify your monitoring configuration, point your web browser at your {kib}
+host, and select **Monitoring** from the side navigation. Metrics reported from
+your Logstash nodes should be visible in the Logstash section. When security is
+enabled, to view the monitoring dashboards you must log in to {kib} as a user
+who has the `kibana_user` and `monitoring_user` roles.
++
+image::images/monitoring-ui.png["Monitoring",link="monitoring/images/monitoring-ui.png"]
+
+include::../settings/monitoring-settings-legacy.asciidoc[]
+