[Metricbeat] Add vpc metricset for aws module #16111

kaiyan-sheng · 2020-02-05T17:14:21Z

What does this PR do?

This PR is to add support for collecting vpc related monitoring metrics from CloudWatch. This includes AWS/NATGateway, AWS/VPN and AWS/TransitGateway namespaces.

Why is it important?

Users can use this vpc metricset to collect metrics related to their VPCs, for example: bytes received/sent through VPN tunnel, total number of concurrent active TCP connections through NAT gateway, and number of bytes sent/received from transit gateway.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

generate data.json
dashboard

How to test this PR locally

Make sure the AWS account you use for testing has VPC related metrics first.
Enable vpc metricset and then change aws.yml to:

- module: aws
  period: 1m
  credential_profile_name: elastic-test
  metricsets:
    - vpc

Related issues

#14854

kaiyan-sheng · 2020-02-05T17:18:43Z

Question: there are three namespaces included in this metricset, should they be split into 3 separate metricsets?
My opinion is they are all VPC related, we can keep them together. But the concern is, by looking at the output events, metricset.name is vpc. But actual metrics are aws.natgateway.metrics.*, aws.transitgateway.metrics.* and aws.vpn.metrics.*.

mtojek · 2020-02-14T07:25:22Z

Question: there are three namespaces included in this metricset, should they be split into 3 separate metricsets?
My opinion is they are all VPC related, we can keep them together. But the concern is, by looking at the output events, metricset.name is vpc. But actual metrics are aws.natgateway.metrics.*, aws.transitgateway.metrics.* and aws.vpn.metrics.*.

I looked into the AWS docs to see if there're mentioned under a common VPC umbrella or separately. Is it possible to add a common key to these metrics? e.g. aws.vpc.vpn.metrics.*, aws.vpc.natgateway.metrics.*

mtojek · 2020-02-14T07:43:39Z

x-pack/metricbeat/module/aws/vpc/vpc_test.go

+import (
+	"os"
+
+	"github.com/elastic/beats/metricbeat/mb"


Shouldn't that be together with other beats imports?

The other two beats import are for registering input module and metricset 😄 So I left them separate with a comment.

kaiyan-sheng · 2020-02-14T14:27:52Z

I looked into the AWS docs to see if there're mentioned under a common VPC umbrella or separately. Is it possible to add a common key to these metrics? e.g. aws.vpc.vpn.metrics.*, aws.vpc.natgateway.metrics.*

Thanks for your review!! Here is the AWS doc that mentioned these three namespaces under VPC: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html
Hmm since this metricset is using light module to create, vpn/natgateway/transitgateway are from the given namespace. Similarly, for ELBs, metrics are started with aws.elb, aws.applicationelb and aws.networkelb.

* Add vpn into vpc metricset * Add NATGateway and TrasitGateway

* Add vpn into vpc metricset * Add NATGateway and TrasitGateway (cherry picked from commit 96100df)

* Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (#16658) * Make memqueue.Broker internal (now memqueue.broker) (#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584) * [Metricbeat] Add vpc metricset for aws module (#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (#16497) * Added NetBeat (#12612) * Close files earlier when generating fields files (#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>

* Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (elastic#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (elastic#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (elastic#16658) * Make memqueue.Broker internal (now memqueue.broker) (elastic#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (elastic#16584) * [Metricbeat] Add vpc metricset for aws module (elastic#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (elastic#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (elastic#16497) * Added NetBeat (elastic#12612) * Close files earlier when generating fields files (elastic#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (elastic#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> (cherry picked from commit 07b03a7)

) * Add cloudfoundry module to metricbeat (#16671) * Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (#16658) * Make memqueue.Broker internal (now memqueue.broker) (#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584) * [Metricbeat] Add vpc metricset for aws module (#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (#16497) * Added NetBeat (#12612) * Close files earlier when generating fields files (#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> (cherry picked from commit 07b03a7) * Fix changelog. * Run mage fmt. * Fix imports to have v7. * Run mage fmt update. * Fix changelog.

kaiyan-sheng · 2020-03-23T15:16:20Z

This will be tested by #16892.

kaiyan-sheng added 2 commits February 5, 2020 07:26

Add vpn into vpc metricset

0a392be

Add NATGateway and TrasitGateway

a286a74

kaiyan-sheng self-assigned this Feb 5, 2020

kaiyan-sheng added enhancement Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team labels Feb 5, 2020

run mage update under beats/metricbeat

7db5102

kaiyan-sheng added the [zube]: In Progress label Feb 6, 2020

kaiyan-sheng marked this pull request as ready for review February 11, 2020 16:03

kaiyan-sheng requested a review from a team as a code owner February 11, 2020 16:03

kaiyan-sheng added 3 commits February 11, 2020 19:27

Merge remote-tracking branch 'upstream/master' into vpc_mb

70227f6

update fields.go

f65c3ca

add changelog

4ac4e18

mtojek approved these changes Feb 14, 2020

View reviewed changes

kaiyan-sheng merged commit 96100df into elastic:master Feb 27, 2020

kaiyan-sheng deleted the vpc_mb branch February 27, 2020 22:05

zube bot added [zube]: Done and removed [zube]: In Progress labels Feb 27, 2020

kaiyan-sheng mentioned this pull request Feb 27, 2020

Cherry-pick #16111 to 7.x: [Metricbeat] Add vpc metricset for aws module #16677

Merged

7 tasks

kaiyan-sheng added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Feb 27, 2020

blakerouse pushed a commit to blakerouse/beats that referenced this pull request Feb 28, 2020

[Metricbeat] Add vpc metricset for aws module (elastic#16111)

6e79791

* Add vpn into vpc metricset * Add NATGateway and TrasitGateway

kaiyan-sheng added a commit that referenced this pull request Feb 28, 2020

[Metricbeat] Add vpc metricset for aws module (#16111) (#16677)

2f48b2d

* Add vpn into vpc metricset * Add NATGateway and TrasitGateway (cherry picked from commit 96100df)

andresrc removed the [zube]: Done label Mar 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add vpc metricset for aws module #16111

[Metricbeat] Add vpc metricset for aws module #16111

kaiyan-sheng commented Feb 5, 2020 •

edited

Loading

kaiyan-sheng commented Feb 5, 2020

mtojek commented Feb 14, 2020

mtojek Feb 14, 2020

kaiyan-sheng Feb 14, 2020

kaiyan-sheng commented Feb 14, 2020

kaiyan-sheng commented Mar 23, 2020

[Metricbeat] Add vpc metricset for aws module #16111

[Metricbeat] Add vpc metricset for aws module #16111

Conversation

kaiyan-sheng commented Feb 5, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

kaiyan-sheng commented Feb 5, 2020

mtojek commented Feb 14, 2020

mtojek Feb 14, 2020

Choose a reason for hiding this comment

kaiyan-sheng Feb 14, 2020

Choose a reason for hiding this comment

kaiyan-sheng commented Feb 14, 2020

kaiyan-sheng commented Mar 23, 2020

kaiyan-sheng commented Feb 5, 2020 •

edited

Loading