[Doc]Restructure monitoring docs to support new and legacy internal collectors #11714

karenzone · 2020-03-24T01:41:58Z

This PR starts with the goodness of #11586, pulls in doc architecture changes from #11614, and resolves recently introduced conflicts.

PREVIEW: http://logstash_11714.docs-preview.app.elstc.co/guide/en/logstash/master/configuring-logstash.html

…nabled setting

…xed all the `xpack.monitoring.*` references

docs/static/settings/monitoring-settings.asciidoc

docs/static/monitoring/collectors.asciidoc

andsel · 2020-04-09T15:30:24Z

LGTM

robbavey

Looks good so far! Couple of nitpicks and suggestions

robbavey · 2020-04-09T15:02:44Z

docs/static/monitoring/collectors.asciidoc

-`xpack.monitoring.collection.enabled` setting on Logstash. You must use the 
-`xpack.monitoring.enabled` setting to enable and disable data collection. 
+WARNING: Unlike for {es} and {kib} monitoring, there is no 
+`pack.monitoring.collection.enabled` setting on Logstash. You must use the


robbavey · 2020-04-09T15:03:31Z

docs/static/monitoring/monitoring-internal-legacy.asciidoc

+See <<monitoring-settings, Monitoring Settings>> for the list of settings.
+
+If you don’t have an Elasticsearch output plugin configured in the pipelines,
+add the setting `monitoring.cluster_uuid` to your logstash.yml.


Do we need to give an explanation of what this should be, and how they can retrieve it?

Not sure how much we should invest in the legacy component as it is no longer one of the preferred approaches. Let's discuss.

True, although we could just use the description from the newer mechanism

robbavey · 2020-04-09T15:05:56Z

docs/static/monitoring/monitoring-internal-legacy.asciidoc

@@ -0,0 +1,152 @@
+[role="xpack"]
+[[monitoring-internal-collection-legacy]]
+=== Collect {ls} monitoring data using legacy internal collectors


Do we have a description of what is meant by 'legacy'?

We have notes at key entry points directing users to one of the other options. What are you recommending?

robbavey · 2020-04-09T15:08:47Z

docs/static/monitoring/monitoring-output.asciidoc

 ---------------------------------------------------

-All data produced by {monitoring} for Logstash is indexed in the monitoring 
+All data produced by Logstash monitoring is indexed in the monitoring 


Need to verify whether this is still correct

robbavey · 2020-04-09T15:09:10Z

docs/static/monitoring/monitoring-output.asciidoc

-This is particularly important to remember when you move from development 
-environments to production environments, where you often have dedicated 
-monitoring clusters.
+IMPORTANT: When discussing security relative to the `elasticsearch` output,


I don't think this is still correct as we don't route through production clusters any more

robbavey · 2020-04-09T15:20:44Z

docs/static/settings/monitoring-settings.asciidoc


+The universally unique identifier (UUID) for the monitoring cluster. 
+By default, {ls} identifies and uses the `cluster uuid` from the first


It uses cluster_uuids from each of the elasticsearch outputs, rather than just the first

robbavey · 2020-04-09T15:30:56Z

docs/static/settings/monitoring-settings.asciidoc


 Optional settings that provide the paths to the Java keystore (JKS) to validate
 the client’s certificate.

-`xpack.monitoring.elasticsearch.ssl.keystore.password`::
+`monitoring.elasticsearch.ssl.keystore.password`::

 Optional settings that provide the password to the keystore.



Also need to change verification_mode

robbavey · 2020-04-09T15:32:22Z

docs/static/settings/monitoring-settings.asciidoc

+
+Finds all {es} cluster nodes and adds them to the hosts list.
+
+Defaults to `false`.


Do you need the newline here? It renders slightly weirdly

docs/static/monitoring/collectors.asciidoc

robbavey · 2020-04-14T19:04:38Z

docs/static/monitoring/monitoring-internal.asciidoc

+These pieces live outside of the default Logstash pipeline in a dedicated
+monitoring pipeline. This configuration ensures that all data and processing has
+a minimal impact on ordinary Logstash processing. Existing Logstash features,
+such as the <<plugins-outputs-elasticsearch,`elasticsearch` output>>, can be


I know this comes from previously written copy - but I'm not sure what it is trying to say. I think it is trying to talk about the reuse of certain features from the elasticsearch output, such as retry, but I'm not 100%

robbavey · 2020-04-14T19:41:45Z

@karenzone - I'm fine with committing this as is, and then work on the fine details in a subsequent PR

karenzone · 2020-04-14T19:47:16Z

This PR is getting crushed under its own weight. I will merge what we have here, and immediately open a new one to track final comments.

Work continues in #11789

…ollectors (elastic#11714) * [Doc] added description of xpack.monitoring.collection.write_direct.enabled setting * Added page to mark as deprecated the legacy internal collector and fixed all the `xpack.monitoring.*` references * Included legacy collector file into monitoring overview * Restructure monitoring docs * Incorporate review comments Co-authored-by: andsel <selva.andre@gmail.com>

…tors (#11714) * [Doc] added description of xpack.monitoring.collection.write_direct.enabled setting * Added page to mark as deprecated the legacy internal collector and fixed all the `xpack.monitoring.*` references * Included legacy collector file into monitoring overview * Restructure monitoring docs * Incorporate review comments Co-authored-by: andsel <selva.andre@gmail.com> Fixes #11787

andsel and others added 4 commits February 17, 2020 14:47

[Doc] added description of xpack.monitoring.collection.write_direct.e…

bfd7a04

…nabled setting

Added page to mark as deprecated the legacy internal collector and fi…

8e07674

…xed all the `xpack.monitoring.*` references

Included legacy collector file into monitoring overview

ffc4ba4

Resolve conflicts

c2d107b

karenzone added the docs label Mar 24, 2020

karenzone mentioned this pull request Mar 24, 2020

[Doc]Rework monitoring docs #11614

Closed

karenzone added work in progress v7.7.0 labels Mar 24, 2020

karenzone force-pushed the monitoring-take3 branch 2 times, most recently from 9b62f01 to a378ad4 Compare March 24, 2020 18:45

andsel reviewed Mar 25, 2020

View reviewed changes

docs/static/settings/monitoring-settings.asciidoc Outdated Show resolved Hide resolved

karenzone force-pushed the monitoring-take3 branch from a378ad4 to 99538db Compare March 27, 2020 21:25

karenzone force-pushed the monitoring-take3 branch 3 times, most recently from f4f3b23 to 19ca70e Compare April 8, 2020 19:24

Restructure monitoring docs

8884fb7

karenzone force-pushed the monitoring-take3 branch from 19ca70e to 8884fb7 Compare April 8, 2020 20:08

karenzone requested review from andsel and robbavey April 9, 2020 14:38

karenzone removed the work in progress label Apr 9, 2020

robbavey reviewed Apr 9, 2020

View reviewed changes

docs/static/monitoring/collectors.asciidoc Show resolved Hide resolved

andsel approved these changes Apr 9, 2020

View reviewed changes

roaksoax assigned roaksoax and acchen97 Apr 9, 2020

robbavey requested changes Apr 9, 2020

View reviewed changes

karenzone force-pushed the monitoring-take3 branch 2 times, most recently from eb5670d to b11f6cb Compare April 14, 2020 14:35

Incorporate review comments

7a5e64e

karenzone force-pushed the monitoring-take3 branch from b11f6cb to 7a5e64e Compare April 14, 2020 14:39

robbavey reviewed Apr 14, 2020

View reviewed changes

robbavey approved these changes Apr 14, 2020

View reviewed changes

karenzone merged commit 081ec78 into elastic:master Apr 14, 2020

karenzone deleted the monitoring-take3 branch April 14, 2020 19:48

karenzone mentioned this pull request Apr 14, 2020

[Doc]Restructure monitoring to support new and legacy internal collectors for 7.x #11787

Closed

karenzone mentioned this pull request Apr 14, 2020

[Doc]Doc updates for internal collectors #11789

Merged

This was referenced Apr 16, 2020

[Meta Issue] Internal collector adjustments for 7.7 #11573

Closed

[Doc] added description of monitoring setting write_direct #11586

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Doc]Restructure monitoring docs to support new and legacy internal collectors #11714

[Doc]Restructure monitoring docs to support new and legacy internal collectors #11714

karenzone commented Mar 24, 2020 •

edited

Loading

andsel commented Apr 9, 2020

robbavey left a comment

robbavey Apr 9, 2020

robbavey Apr 9, 2020

karenzone Apr 9, 2020

robbavey Apr 9, 2020

robbavey Apr 9, 2020

karenzone Apr 9, 2020

robbavey Apr 9, 2020

karenzone Apr 9, 2020

robbavey Apr 9, 2020

karenzone Apr 9, 2020

robbavey Apr 9, 2020

robbavey Apr 9, 2020

robbavey Apr 9, 2020

robbavey Apr 14, 2020

robbavey commented Apr 14, 2020

karenzone commented Apr 14, 2020 •

edited

Loading


		The universally unique identifier (UUID) for the monitoring cluster.
		By default, {ls} identifies and uses the `cluster uuid` from the first


		Finds all {es} cluster nodes and adds them to the hosts list.

		Defaults to `false`.

[Doc]Restructure monitoring docs to support new and legacy internal collectors #11714

[Doc]Restructure monitoring docs to support new and legacy internal collectors #11714

Conversation

karenzone commented Mar 24, 2020 • edited Loading

andsel commented Apr 9, 2020

robbavey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robbavey commented Apr 14, 2020

karenzone commented Apr 14, 2020 • edited Loading

karenzone commented Mar 24, 2020 •

edited

Loading

karenzone commented Apr 14, 2020 •

edited

Loading