[Metricbeat] AWS metric start time and end time should align with the value of the metric's period

[kwinstonix]

Sometime metricbeat get empty metric result of EC2 instance. There is an issue in aws.GetStartTimeEndTime

• Version: 8.3.3
• Steps to Reproduce:

get aws ec2 instance metrics with the python script.

import boto3
import datetime

client = boto3.client("cloudwatch", region_name="ap-northeast-2")

period = 300
end = datetime.datetime.utcnow().replace(second=0)
start = end - datetime.timedelta(seconds=period * 1)

response = client.get_metric_data(
    MetricDataQueries=[
        {
            "Id": "id1",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/EC2",
                    "MetricName": "CPUUtilization",
                    "Dimensions": [
                        {"Name": "InstanceId", "Value": "your instance id"},
                    ],
                },
                "Period": period,
                "Stat": "Average",
            },
            "Label": "label1",
        },
    ],
    StartTime=start,
    EndTime=end,
)

print('now: ', datetime.datetime.utcnow())
print('end: ', end)
print(response["MetricDataResults"])

If end mod period is between 60s and 120s , there may be empty metric result.

now:  2022-08-09 13:51:12.032382
end:  2022-08-09 13:51:00.150606
[{'Id': 'id1', 'Label': 'label1', 'Timestamps': [], 'Values': [], 'StatusCode': 'Complete'}]

now:  2022-08-09 13:51:03.090935
end:  2022-08-09 13:51:00.091199
[{'Id': 'id1', 'Label': 'label1', 'Timestamps': [], 'Values': [], 'StatusCode': 'Complete'}]

• CloudWatch metricset use this function to get start and end timestamp.

beats/x-pack/metricbeat/module/aws/utils.go

Lines 22 to 37 in 62ddcc4

	func GetStartTimeEndTime(period time.Duration, latency time.Duration) (time.Time, time.Time) {
	endTime := time.Now()
	if latency != 0 {
	// add latency if config is not 0
	endTime = endTime.Add(latency * -1)
	}
	// Set startTime to be one period earlier than the endTime. If metrics are
	// not being collected, use latency config parameter to offset the startTime
	// and endTime.
	startTime := endTime.Add(period * -1)
	// Defining duration
	d := 60 * time.Second
	// Calling Round() method
	return startTime.Round(d), endTime.Round(d)
	}

• According to the AWS API doc, we should make StartTime and EndTime align with period.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html#API_GetMetricData_RequestParameters

For better performance, specify StartTime and EndTime values that align with the value of the metric's Period and sync up with the beginning and end of an hour. For example, if the Period of a metric is 5 minutes, specifying 12:05 or 12:30 as StartTime can get a faster response from CloudWatch than setting 12:07 or 12:29 as the StartTime.

• How to fix this

-    endTime := time.Now()
+    endTime := time.Now().Truncate(period)

[kwinstonix]

@kaiyan-sheng , can you take a look at this issue?