-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Metricbeat] Add aggregation aligner as a config param for stackdriver metricset in GCP #17719
[Metricbeat] Add aggregation aligner as a config param for stackdriver metricset in GCP #17719
Conversation
Pinging @elastic/integrations-platforms (Team:Platforms) |
I like the new approach! is it working as expected? The PR description will need to be updated |
|
||
//stackDriverConfig holds a configuration specific for stackdriver metricset. | ||
type stackDriverConfig struct { | ||
MetricTypes []string `config:"metric_types" validate:"required"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these are not actually types but metric names right? how about using names
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmmmm to be consistent with GCP, it is metric.type
. metric.type
is a part of metric selector, which includes its DNS name prefix. Here is an exmaple:
metric.type = "compute.googleapis.com/instance/cpu/usage_time" AND
metric.labels.instance_name = "my-instance-name"
CI failures are not seem to be related. |
What does this PR do?
aligner
config parameter for stackdriver metriset under metrics.cpu.utilization.avg
forALIGN_MEAN
,cpu.utilization.sum
forALIGN_SUM
andcpu.utilization.value
forALIGN_NONE
.metricbeat/docs/fields.asciidoc
to include mappings fromgooglecloud
module.data-*.json
files that are not generated by integration test TestData for each metricset.Why is it important?
ListMetricDescriptors API is used to get metadata sample period and ingest delay for each metric type once at the start of this module. If sample period is smaller than the collection period, aggregation will be used in
ListTimeSeries
API. By default, aligner isALIGN_NONE
. This means if user specify this Metricbeat collection period to be5m
and the metric type sample period is 60s, then Metricbeat will return 5 raw data points (1 for each minute) in one ListTimeSeries API call. This will save cost significantly if user does not mind the extra delay. If user wants to only return one aggregated metric per collection period, aligner can be specified, such asALIGN_MEAN
,ALIGN_SUM
and etc.Monitoring collects one measurement each minute (the sampling rate), but it can take up to 4 minutes before you can retrieve the data (latency). In order to make sure the collection is successful, we delay collection startTime and endTime for a number of minutes defined by
ingest delay
every time. Instead of hardcodingingest delay
to 4 minute, this is obtained fromListMetricDescriptors
API for each metric type.Assume
ingest delay = 4-minute
,sample period = 1-minute
andcollection period = 1-minute
, when querying GCP APItimeSeries.list
, the time interval changed to:Therefore, data collection will always have a delay. This is consistent with monitoring in GCP portal.
Assume
ingest delay = 4-minute
,sample period = 5-minute
, aggregation aligner isALIGN_MEAN
andcollection period = 5-minute
, when querying GCP APItimeSeries.list
, the time interval changed to:Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.How to test this PR locally
Two test cases here:
Use config below and you should see 5 metrics every 5 minutes:
and output event looks like this:
Use config below and you should see 1 metric every 1 minute:
and output event looks like this:
Related issues
TODOs
This PR is getting too big so I will list things need to be done in separate PRs:
Investigate distribution type value in google cloud:
beats/x-pack/metricbeat/module/googlecloud/stackdriver/response_parser.go
Line 99 in 95626b8
change aligner to
aligners
for a list of stringsmove
service
intometrics
config and change config to look like this:data.json
files for different metric types inside each metricset.