CloudWatch: High resolution is not work

[mtanda]

What happened:
High resolution query cause error.
The error message is too many datapoint requested in query A. Please try to reduce the time range.

What you expected to happen:
Get success response.

How to reproduce it (as minimally and precisely as possible):

• Check high resolution
• Set time range to 30 days
• Select 12 or more dimensions (the expected response size should exceed 100,800 data points)
  • If test with AWS/EC2 CPUUtilization (the metrics resolution is 5 min), 100800 / ((60 / 5) * 24 * 30) = 11.6, select 12 InstanceId

Anything else we need to know?:
Original implementation is #10925 .

Environment:

• Grafana version: v6.5.0 beta-1
• Data source type & version: CloudWatch
• OS Grafana is installed on: Mac
• User OS & Browser: Firefox
• Grafana plugins:
• Others:

[hugohaggmark]

Thanks @mtanda for reporting this!

[sunker]

Thanks for reporting this @mtanda.

I'm afraid I don't see how we could support HighRes now that we use GMD instead of GMS. The reason is that even though there's plenty of queries in the panel, we only make one call to GetMetricData API. That call uses the same startTime and endTime for all queries, so really there's no way to define HighRes on a per query level anymore. Or do you have any solution to this @mtanda? That is something that does not include multiple GMD calls.

So the error message that you've received is expected - a workaround is to reduce the timespan. I suggest I remove the HighRes toggle from the GUI and also describe this as a breaking change in the changelog.

[sunker]

@mtanda I'm also available on grafanas public slack if you'd prefer to discuss this over there.

[mtanda]

I send a partial fix of this issue.
#20459

If the query doesn't include expression, datasource can handle high resolution.

[mtanda]

The hardest limitation of CloudWatch API is, NextToken is not returned when query has expression.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDataResult.html

The status of the returned data. Complete indicates that all data points in the requested time range were returned. PartialData means that an incomplete set of data points were returned. You can use the NextToken value that was returned and repeat your request to get more data points. NextToken is not returned if you are performing a math expression. InternalError indicates that an error occurred. Retry your request using NextToken, if present.

New GMD implementation utilize SEARCH() expression for expanding template variable.
So, I think only solution is splitting original query to multiple SEARCH() expression query...

@sunker