New component: AWS CloudWatch metrics receiver

[lewis262626]

The purpose and use-cases of the new component

Collect metrics from AWS Cloudwatch metrics and parse them in to OTel pmetric. Some metrics on AWS, such as Lambda memory usage etc, are only available from CloudWatch — users may want them to be put in one platform for analysis such as Prometheus with Grafana.

Example configuration for the component

receivers:
    awscloudwatchmetrics:
        region: <aws_region>
        profile: <aws_profile> # optional if omitted will use default for current if the AWS credential environment variables are not set
        polling_interval: 60
        metrics:
          named: 
             - namespace: AWS/ALB
                metric_name: RequestCount
             - namespace: AWS/ALB
                metric_name: 500Count

Telemetry data types supported

Metrics

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

No response

Additional context

Functionality is similar to AWSContainerMetrics receiver, maybe extend that instead and make it more generic?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[jpkrohling]

On behalf of the OpenTelemetry Squad at Grafana Labs, I would like to volunteer as a sponsor for this component. In addition to the component's author, the code owners (as sponsors) will be:

• @jpkrohling
• @kovrus

While we would very much prefer the affected vendor to support this receiver, we understand they might not be interested in translating their proprietary formats into OTLP, only the other way around.

We believe that the community wants this receiver, and if community members are willing to contribute code for this, we are willing to support the community.

[lewis262626]

@jpkrohling That's great, more than happy with that approach. Thanks to Grafana for stepping up :)

[jpkrohling]

@lewis262626, once you submit the PR, let me know. Please, make sure to follow the contributing guidelines for new components.

[lewis262626]

Sorry for the delay -- have made a start on this :)

[lewis262626]

@evan-bradley Please could I be assigned this issue

[matej-g]

Hi @lewis262626, has any work been started on this? We're also interested in having this receiver in the collector, happy to help where needed (contributing PRs, review).

[secustor]

Has it been considered to implement this as part of awscloudwatchreceiver?

awscloudwatchreceiver has all configuration options under a logs attribute, so it make sense to implement it their and add the configuration options for metrics simply under a separate metrics key.

For reference there is also an older issue regarding this component: #6226

[lewis262626]

@matej-g H Yes. You can view my progress under my fork https://github.com/lewis262626/opentelemetry-collector-contrib/tree/awscloudwatchmetricsreceiver/receiver/awscloudwatchmetricsreceiver

I only have an hour or so a week to commit to this so been slow progress I'm afraid. I have:

• Implement config for receiver
• Add unit tests to verify config parsing
• Add newFactory method
• Add unit tests for factory code
• Implement AWS API calls (am going to use GetMetricData)
• Parse in to pmetrics
• Add unit tests for pmetrics conversion logic

Any help much appreciated

[lewis262626]

@secustor Potentially, yep. Although other metric receivers are standalone such as the awscontainerinsightreceiver

[matej-g]

@lewis262626 Thanks for sharing that. Let me know what form of help would make it possible to move this forward. Happy to take that code over if you're busy or feel free to open a draft PR if you'd like.

[lewis262626]

@matej-g I'll open a draft PR then

[lewis262626]

I've been pretty much done all the logic. Now onto finishing off the unit tests and integration tests.

Should I also implement this logic or leave for a later version?

The time stamp indicating the earliest data to be returned.
The value specified is inclusive; results include data points with the specified time stamp.
CloudWatch rounds the specified time stamp as follows:
Start time less than 15 days ago - Round down to the nearest whole minute. For example, 12:32:34 is rounded down to 12:32:00.
Start time between 15 and 63 days ago - Round down to the nearest 5-minute clock interval. For example, 12:32:34 is rounded down to 12:30:00.
Start time greater than 63 days ago - Round down to the nearest 1-hour clock interval. For example, 12:32:34 is rounded down to 12:00:00.
If you set Period to 5, 10, or 30, the start time of your request is rounded down to the nearest time that corresponds to even 5-, 10-, or 30-second divisions of a minute. For example, if you make a query at (HH:mm:ss) 01:05:23 for the previous 10-second period, the start time of your request is rounded down and you receive data from 01:05:10 to 01:05:20. If you make a query at 15:07:17 for the previous 5 minutes of data, using a period of 5 seconds, you receive data timestamped between 15:02:15 and 15:07:15.
For better performance, specify StartTime and EndTime values that align with the value of the metric's Period and sync up with the beginning and end of an hour. For example, if the Period of a metric is 5 minutes, specifying 12:05 or 12:30 as StartTime can get a faster response from CloudWatch than setting 12:07 or 12:29 as the StartTime .

[matej-g]

Hey @lewis262626,
Thanks for opening this up. I took a look at the PR, it looks like a good start. I wonder though do you intend to implement the actual polling?

As I mentioned previously, I'd be happy to take it from there if you're busy.

[lewis262626]

Hey @lewis262626,

Thanks for opening this up. I took a look at the PR, it looks like a good start. I wonder though do you intend to implement the actual polling?

As I mentioned previously, I'd be happy to take it from there if you're busy.

Hey, I'll reply to your PR comments in a bit. To answer your question, the receiver is done and I've been using to pull metrics from CloudWatch.

I was asked to split my PR, but if you want to use my code feel free to check out my branch as referenced in #19218.

[matej-g]

Got it, thanks @lewis262626, in that case I'll wait on that second PR.

[dthauvin]

Hi @wesleyreis1808 , does this receiver can handle EKS IRSA ( IAM ROLE for Service Account) and find credential with this Env variable :

AWS_DEFAULT_REGION : eu-west-1
AWS_REGION : eu-west-1
AWS_ROLE_ARN : arn:aws:iam::xxxxxxxxxxxxx:role/myrole
AWS_STS_REGIONAL_ENDPOINTS : regional
AWS_WEB_IDENTITY_TOKEN_FILE : /var/run/secrets/eks.amazonaws.com/serviceaccount/token

I do not find any trace of usage GetMetricData api in AWS cloudtrail , and i do not find any debug usefull information.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.