-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to override cluster_uuid to be used in monitoring data #11066
Comments
Pinging @elastic/stack-monitoring. |
This is mostly done in this PR: #11066 I am blocked on a test issue that I can't get past and I am waiting for the LS to have a little time to look at it. I see that it was just recently rebased, however, so they may be close on this. |
@cachedout PR #11106 which exposes |
@andsel Thanks! I think it's best to let @ycombinator look over this before we close it and see if there are any outstanding issues to be resolved. Thanks again for your work! |
Closed via #11106. |
Background and Problem
Logstash monitoring data is stored in
.monitoring-logstash-*
indices for the Stack Monitoring application. A key field in these documents iscluster_uuid
. The value of this field is supposed to be an Elasticsearch cluster UUID. The value determines where in the Stack Monitoring UI the Logstash monitoring data shows up, i.e. which Elasticsearch cluster the data is grouped with.Starting 7.3.0, the Metricbeat
logstash
module is able to interrogate various Logstash APIs to collect monitoring data required by Stack Monitoring. Metricbeat builds Elasticsearch documents from this data, intended to be indexed into.monitoring-logstash-*
indices.If a Logstash pipeline is using an Elasticsearch output, that Elasticsearch cluster's UUID is returned in the pipeline vertex object for the output in Logstash API responses. Metricbeat uses this cluster UUID value as the value of the
cluster_uuid
field in documents it indexes into.monitoring-logstash-*
.However, if a Logstash pipeline is not using an Elasticsearch output, Metricbeat will not index any value for the
cluster_uuid
field. As a result the Logstash monitoring data will show up under a Standalone Cluster in the Stack Monitoring UI.A similar thing happens for Beats monitoring data as well, and users are not happy about it. Such users would like to see their monitoring data associated with their production Elasticsearch cluster in the UI.
Knowing how users have responded to Beats monitoring data showing up under Standalone Cluster in the Stack Monitoring UI, wo we'd like to get ahead of this problem for Logstash monitoring data.
Suggested Solution
In the same vein as elastic/beats#13182, add a new optional setting to
logstash.yml
namedxpack.monitoring.cluster_uuid
.Any Logstash APIs that return pipeline representations (vertices and edges) would be affected by this setting. If this setting is set, such APIs should also return a top-level
cluster_uuid
property.The
logstash
Metricbeat module will first look for the top-levelcluster_uuid
property. If it is present, it will use its value as the value of thecluster_uuid
field in.monitoring-logstash-*
documents. If its not present, the module will fallback to it's current behavior.The text was updated successfully, but these errors were encountered: