Add Metricbeat AWS EC2 Module

[kaiyan-sheng]

This PR is to add AWS EC2 metricset into metricbeat. This PR enables metricbeat to periodically fetch EC2 monitoring metrics from AWS Cloudwatch using GetMetricData API. Also for each EC2 instance, it uses DescribeInstances API to get detailed information about a specific instance.

Checklist:

• Integration tests exist
• System tests exist
• Automated checks that all fields are documented
• Documentation
• Fields follow ECS and naming conventions
• Example data.json exists and an automated way to generate it exists (go test -data)
• Dashboards exists (if applicable)
• Kibana Home Tutorial (if applicable)

There are 16 cloudwatch metrics for EC2 monitoring collected by AWS EC2 module:
CPUUtilization -> aws.ec2.cpu.total.pct
CPUSurplusCreditBalance -> aws.ec2.cpu.surplus_credits_balance
CPUSurplusCreditsCharged -> aws.ec2.cpu.surplus_credits_charged
CPUCreditBalance -> aws.ec2.cpu.credit_balance
CPUCreditUsage -> aws.ec2.cpu.credit_usage
NetworkPacketsIn -> aws.ec2.network.in.packets
NetworkPacketsOut -> aws.ec2.network.out.packets
NetworkIn -> aws.ec2.network.in.bytes
NetworkOut -> aws.ec2.network.out.bytes
DiskReadBytes -> aws.ec2.diskio.read.bytes
DiskWriteOps -> aws.ec2.diskio.write.bytes
DiskReadOps -> aws.ec2.diskio.read.ops
DiskWriteBytes -> aws.ec2.diskio.write.ops
StatusCheckFailed -> aws.ec2.status.check_failed
StatusCheckFailed_System -> aws.ec2.status.check_failed_system
StatusCheckFailed_Instance -> aws.ec2.status.check_failed_instance

Also this PR adds metric collection for basic AWS EC2 instance information:
cloud.provider
cloud.instance.id
cloud.image.id
cloud.machine.type
cloud.region
cloud.availability_zone

[kaiyan-sheng]

I'm thinking by following https://github.com/elastic/ecs#network, rename the cpu related metrics to cpu.*, network related metrics to network.*, disk related metrics to disk.*, and status related metrics to status.*.

[ruflin]

For the metrics best follow the naming scheme that the system metricbeat module uses.

[kaiyan-sheng]

@ruflin Yes good point, but some of the cloudwatch metrics don't have a direct match to metrics in system metricbeat module. What I found can be a direct map are:

• cpu_utilization -> cpu.total.pct
• network_packets_in -> network.in.packets
• network_packets_out -> network.out.packets
• network_in -> network.in.bytes
• network_out -> network.out.bytes
• disk_read_bytes -> diskio.read.bytes
• disk_write_bytes -> diskio.write.bytes

What do you think?

[ruflin]

Yes, lets only map the ones that exist, the other ones which are EC2 specific I would keep prefixed.

Some of the above values are actually matching to ECS which is even nicer: https://github.com/elastic/ecs#network For the cpu / disk ones: Keep in mind that they are prefixed by system.*

[exekias]

This is looking great @kaiyan-sheng 🎉 🎉. I left a few comments, nothing major.

I'm not very comfortable with the need of a mock config param for tests, will try to think of alternatives to that and propose them here 😇

[exekias]

@ruflin about metric names, as there is no direct map with system module, I think that the current approach (names like aws.ec2.cpu.total.pct) is correct. As a user that will help me find ec2 related metrics while typing them in the bar.

At some point we will also want this to work with current Waffle map view and others, let's review the metrics that are being used there and see our options. For instance, we could put those under system.* namespace and create an alias to have them under aws.ec2.* too

[ruflin]

@exekias I like the idea to start to play with aliases here. We will have to make sure that the units etc. are all in sync.

[kaiyan-sheng]

@exekias I thought about the mock parameter as well as how should we pass in the aws credentials a little bit more. What I came up with is, using aws_config_profile_name to replace mock and aws credentials. Please let me know if this make sense :-)

aws_config_profile_name is the profile name in local AWS configuration file that the user want this aws metricbeat module to use. If aws_config_profile_name is not set, this module will report mock data instead.

The AWS CLI stores the credentials that you specify with aws configure in a local file named credentials in a folder named .aws in your home directory ~/.aws/credentials. The file generated by the AWS CLI for the profile configured in the previous section look like this:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
[metricbeat-aws]
aws_access_key_id=ASIAZENKQPPNZDBTSVNB
aws_secret_access_key=Saxlp2wI09tMcaBhwkEMtWBjaGUwtrmGE
aws_session_token=FQoGZXIvYXdzEHsaDKpiiMUFzVp16DT6MSKwAfm6efgVZWRcziaBt33Fn1Z3PvPfOOCXm0eWasKJlCenFLKfU2gaTLb0JursUeH+Af6qfJobz7MhpVYNbors7Ct59pOgYMRa8+3E1+3vW2iH7QDUdq/hIpK/kh03EotxMRb

Both default and metricbeat-aws in the example above are profile names. If we specify aws_config_profile_name: "metricbeat-aws", then aws module will go to ~/.aws/credentialsand load config from profile metricbeat-aws.

[exekias]

That's an interesting approach, let's split the testing part from client settings.

Historically our users struggle with external files for configuration, as it requires more orchestration to configure. I think previous config was better as it allowed using config params, and if you want, pass these params through ENV vars. I've found a possible solution to avoid SetEnv and reading from files, I think this should work with the aws package:

awsAccessKey := "Insert Key Here" 
awsSecret := "Insert Secret Here" 
token := ""

creds := credentials.NewStaticCredentials(awsAccessKey, awsSecret, token)

cfg := aws.NewConfig().WithRegion("us-west-1").WithCredentials(creds)

Let me know what you think! I'll have a look to the mock and post again :)

[kaiyan-sheng]

Thanks @exekias for your review!! @ph Just curious, did you use credentials in ~/.aws/credentials or environment variables in functionbeat?

[kaiyan-sheng]

Sounds good. Thanks for testing it and for all the comments. I created a meta issue for this and also added a checklist for this PR. I will merge this PR and create a new PR to address the rest of the comments.