Skip to content

A CloudWatch exporter for Prometheus written in Go, with multi-region and multi-account support

License

Notifications You must be signed in to change notification settings

mtlang/cloudwatch_exporter

 
 

Repository files navigation

Mass CloudWatch Exporter

An AWS CloudWatch exporter for Prometheus coded in Go, with multi-region, multi-account, and dynamic target support.

This fork was made to extend the work done by Technofy's CloudWatch Exporter. The goal of this project is to create a cloudwatch exporter that can easily scale to monitor resources across many AWS accounts and many AWS regions. Where other exporters would require an instance to be run in every account or every region, this project aims to monitor an entire AWS landscape and make all gathered information available from a single endpoint. Where other exporters aim to monitor many metrics within a single account and region, this exporter aims to monitor a small number of metrics across many accounts and regions.

Running the Exporter

To run the exporter, simply download the latest release for your platform. Alternatively, if you have Go installed, you can run the exporter from source. When invoking the exporter, you can specify the following flags:

Flag Name Default Value Description
--web.listen-address :9042 Address on which to expose metrics.
--web.telemetry-path /metrics Path under which to expose exporter's metrics.
--web.telemetry-scrape-path /scrape Path under which to expose CloudWatch metrics.
--config.file config.yml Path to configuration file.

Configuration

The exporter is configured with a single YAML file. The following demonstrates the structure of the configuration file:

accounts:
 - 'aws_account_number_1'
 - 'aws_account_number_2'
tasks:
  - name: 'unique_task_name'
   region: 'aws_region' or 'all' (Optional)
   account: 'aws_account_number' or 'all' (Optional)
   role_name: 'name_of_role_to_assume' (Optional)
   metrics:
    - aws_namespace: 'cloudwatch_metric_namespace'
      aws_dimensions: ['cloudwatch_metric_dimension_1', 'cloudwatch_metric_dimension_2'] (Optional)
      aws_dimensions_select: (Optional)
        <name_of_dimension>: ['value_of_dimension']
      aws_dimensions_select_regex: (Optional)
        <name_of_dimension>: 'regex_for_value_to_match'
      aws_metric_name: 'cloudwatch_metric_name'
      aws_statistics: ['metric_statistic_1', 'metric_statistic_2']
      range_seconds: length_of_search_window_in_seconds (Defaults to 600)
      period_seconds: aws_metric_period_in_seconds (Defaults to 60)
      delay_seconds: delay_in_seconds (Defaults to 0)

Configuration Fields

At the top level of the configuration file are three fields: accounts, exclude_accounts, and tasks. Accounts is just a list of AWS account numbers. This list is used by tasks that are set to scrape all accounts. If exclude_accounts are specified, any accounts in that list will not be scraped, even if they're in the accounts list.

A task is a group of metrics which you would like to be presented together. Metrics are scraped by task, so only put metrics under the same task if you want them to always be presented together.

In addition to a list of metrics and a unique identifier, each task can also have three optional fields. A 'region' can be specified or set to 'all'. An 'account' number can be specified, or set to 'all' to use the list defined at the top level. If 'account' is specified, you must also specify a 'role_name'. The exporter will attempt to assume the specified role in the specified account to gather metrics. If any of the optional fields are not specified, the default credential chain will be used instead.

Each metric is defined by several fields:

Field Name Type Required? Description
aws_metric_name string Yes Name of the metric.
aws_namespace string Yes The namespace of the metric. Supports custom namespaces.
aws_dimensions list of strings No Dimentions to aggregate metric across. Required for metrics with dimensions.
aws_dimensions_select map No Optional filter. Maps from name of dimension to acceptable values for dimension.
aws_dimensions_select_regex map No Optional filter. Maps from name of dimension to regex for values to match.
aws_statistics list of strings Yes Statistics to display. Doesn't support extended statistics.
aws_extended_statistics list of strings No Extended Statistics to display.
range_seconds number No Length of metric window in seconds.
delay_seconds number No Delays the end of the metric window by x seconds. If 0, ends window at current time.
period_seconds number No Metric period.

The $_target token in the dimensions select is used to pass a parameter given by Prometheus (for example a __meta tag with service discovery).

Example Configuration

accounts:
 - '111111111111'
 - '222222222222'
exclude_accounts:
 - '222222222222'
tasks:
  - name: lambda_errors
   region: 'all'
   account: 'all'
   role_name: 'cloudwatch-reader'
   metrics:
    - aws_namespace: "AWS/Lambda"
      aws_dimensions: ['FunctionName']
      aws_dimensions_select_regex:
        FunctionName: 'custodian-.*'
      aws_metric_name: 'Errors'
      aws_statistics: ['Maximum']
      range_seconds: 900
      period_seconds: 300
      delay_seconds: 0
 - name: billing
   region: us-east-1
   metrics:
    - aws_namespace: "AWS/Billing"
      aws_dimensions: [Currency]
      aws_dimensions_select:
        Currency: [USD]
      aws_metric_name: EstimatedCharges
      aws_statistics: [Maximum]
      range_seconds: 86400

 - name: ec2_cloudwatch
   region: 'all'
   metrics:
    - aws_namespace: "AWS/EC2"
      aws_dimensions: [InstanceId]
      aws_dimensions_select:
        InstanceId: [$_target]
      aws_metric_name: CPUUtilization
      aws_statistics: [Average]
      aws_extended_statistics: ['p99']

    - aws_namespace: "AWS/EC2"
      aws_dimensions: [InstanceId]
      aws_dimensions_select:
        InstanceId: [$_target]
      aws_metric_name: NetworkOut
      aws_statistics: [Average]
      aws_extended_statistics: ['p99']

  - name: vpn_mon
    region: 'all'
    metrics:
     - aws_namespace: "AWS/VPN"
       aws_dimensions: [VpnId]
       aws_dimensions_select:
         VpnId: [$_target]
       aws_metric_name: TunnelState
       aws_statistics: [Average]
       range_seconds: 3600

Hot reload of the configuration

Let's say you can't afford to kill the process and restart it for any reason and you need to modify the configuration on the fly. It's possible! Just call the /reload endpoint.

Endpoints

Endpoint Description
/metrics Gathers metrics from the CloudWatch exporter itself such as the total number of requests made to the AWS CloudWatch API.
/scrape Gathers metrics from the CloudWatch API depending on the task and (optionally) the target passed as parameters.
/reload Does a live reload of the configuration without restarting the process

For example a scrape URL could look like this:

http://localhost:9042/scrape?task=ec2_cloudwatch&target=i-0123456789&region=eu-west-1

The "target" and "region" parameters are optional, but the "task" parameter is required.

How to configure Prometheus

  - job_name: 'aws_billing'
    metrics_path: '/scrape'
    params:
      task: [billing]
    static_configs:
      - targets: ['localhost:9042']

  - job_name: 'ec2_cloudwatch'
    metrics_path: '/scrape'
    ec2_sd_configs:
      - region: eu-west-1
    params:
      region: [eu-west-1]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_role]
        regex: webapp
        action: keep
      - source_labels: [job]
        target_label: __param_task
      - source_labels: [__meta_ec2_instance_id]
        target_label: __param_target
      - target_label: __address__
        replacement: 'localhost:9042'

  - job_name: 'vpn_mon'
    metrics_path: '/scrape'
    params:
      task: [vpn_mon]
    static_configs:
      - targets: ['vpn-aabbccdd']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: 'localhost:9042'

Thanks to Prometheus relabelling feature, in the second job configuration, we tell it to use the job_name as the task parameter and to use the __meta_ec2_instance_id as the target parameter. The region is specified in the params section.

The Billing example is there to demonstrate the multi-region capability of this exporter, the default_region parameter is specified in the exporter's configuration.

Note: It would also work if no default_region was specified but a params block with the region parameter was set in the Prometheus configuration.

End Note

This exporter is largely inspired by the official CloudWatch Exporter and we'd like to thank all the contributors who participated to the original project.

This project is licensed under the Apache 2.0 license.

About

A CloudWatch exporter for Prometheus written in Go, with multi-region and multi-account support

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 89.7%
  • Makefile 8.5%
  • Dockerfile 1.8%