Metricbeat logstash module: accept override cluster UUID from Logstash
#15795
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ycombinator
added
enhancement
in progress
|
Pinging @elastic/stack-monitoring (Stack monitoring) |
andresrc
added
[zube]: Inbox
Team:Services
|
Logstash pipelines are more or less independent entities (unless one forwards between pipelines). Don't we want pipeline 1 be reported with cluster 1, and pipeline 2 with cluster 2? This changes looks like all pipelines will now be reported with cluster 1 only. |
|
@urso Yeah, we can do that. We can invert the logic to let the pipeline's cluster UUID (if set) to override the global one. I'll make the change. Thanks! |
ycombinator
force-pushed
the
mb-ls-xp-cluster-uuid
branch
from
ae540d0
to
e656ba0
Compare
houndci-bot
reviewed
Feb 4, 2020
houndci-bot
reviewed
Feb 4, 2020
|
@urso I've made the change you suggested. Thanks! |
urso
approved these changes
Feb 4, 2020
|
jenkins, test this |
ycombinator
force-pushed
the
mb-ls-xp-cluster-uuid
branch
from
dc9e5de
to
9f62931
Compare
sayden
approved these changes
Feb 5, 2020
There was a problem hiding this comment.
LGTM, tested manually and getting the following output:
➜ metricbeat git:(mb-ls-xp-cluster-uuid) ✗ ./metricbeat | jq -c 'select(.service.type == "logstash") | .type,.cluster_uuid'
"logstash_state"
"foobar"
"logstash_stats"
"foobar"
"logstash_state"
"foobar"
"logstash_state"
"foobar"
"logstash_stats"
"foobar"
"logstash_state"
"foobar"
"logstash_state"
"foobar"
"logstash_state"
"foobar"
ycombinator
added a commit
to ycombinator/beats
that referenced
this pull request
Feb 5, 2020
…ash (elastic#15795) * Fetch override cluster UUID from Logstash node pipelines API and use it * Parse override cluster UUID from Logstash API response and make node_stats use it * Adding CHANGELOG entry * Use override cluster UUID as fallback to ES vertex cluster UUID * Fixing CHANGELOG entry language * Adding godoc for new helper function
ycombinator
added a commit
that referenced
this pull request
Feb 10, 2020
…ash (#15795) (#16105) * Fetch override cluster UUID from Logstash node pipelines API and use it * Parse override cluster UUID from Logstash API response and make node_stats use it * Adding CHANGELOG entry * Use override cluster UUID as fallback to ES vertex cluster UUID * Fixing CHANGELOG entry language * Adding godoc for new helper function
ycombinator
removed
the
needs_backport
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR teaches the
logstashMetricbeat module to accept an optional overridecluster_uuidfield from Logstash APIs called by the module.If the Logstash pipeline being monitored by this module has an Elasticsearch plugin vertex in it, the cluster UUID associated with that vertex will be used. If no such vertex is found, the override cluster UUID will be used.
Why is it important?
See #15772.
Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesHow to test this PR locally
Configure Logstash to use an override cluster_uuid for monitoring. To do this, add this line to your
config/logstash.yml:Start a simple pipeline in Logstash.
Enable the
logstash-xpackmodule in Metricbeat. This will enable thelogstashmodule with the required configuration for Stack Monitoring.For easy inspection of generated events, disable the
elasticsearchoutput and enable theconsoleoutput in Metricbeat. To do this, edit yourmetricbeat.ymlfile and setoutput.elasticsearch.enabled: falseandoutput.console.enabled: true.Run Metricbeat and check that the generated Logstash Stack Monitoring events of
type:logstash_stateandtype:logstash_statsboth have a top-level field ofcluster_uuidwith the same value as set in the Logstash configuration file.Related issues