Add metrics for ec2 api calls made by CNI and expose via prometheus

[jaydeokar]

What type of PR is this? feature

Which issue does this PR fix: #2035

What does this PR do / Why do we need it:
We don't collect metrics about the number of calls made to EC2 APIs. This change adds the capability to track metrics about the number of calls made to EC2 API by IPAMD. cni-metrics-helper is configured to collect these metrics and push it to cloudwatch

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:
N/A

Testing done on this change:
Yes this has been tested on EKS 1.23 cluster.

# HELP awscni_ec2api_req_count The number of requests made to EC2 APIs by CNI
# TYPE awscni_ec2api_req_count counter
awscni_ec2api_req_count{fn="AssignPrivateIpAddresses"} 1
awscni_ec2api_req_count{fn="AttachNetworkInterface"} 1
awscni_ec2api_req_count{fn="CreateNetworkInterface"} 1
awscni_ec2api_req_count{fn="DeleteNetworkInterface"} 1
awscni_ec2api_req_count{fn="DescribeInstances"} 1
awscni_ec2api_req_count{fn="DescribeNetworkInterfaces"} 13
awscni_ec2api_req_count{fn="DetachNetworkInterface"} 1
awscni_ec2api_req_count{fn="ModifyNetworkInterfaceAttribute"} 2

Automation added to e2e:
N?A

Will this PR introduce any new dependencies?:
N/A

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
No breaking changes. Yes tested on EKS 1.23 cluster

Does this change require updates to the CNI daemonset config files to work?:
Yes, updated image names for CNI and cni metrics helper image tags

Does this PR introduce any user-facing change?:
Yes

- CNI metrics helper can now report the number of EC2 API calls and number of failed calls and visualize it in cloud watch

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jdn5126]

Changes look good to me overall. Two high-level questions:

1. How did you make sure that you did not miss any EC2 calls? Did you just grep on all of the EC2 wrapper functions?
2. Do you think it makes sense to also count EC2 errors? I wonder if we should additionally count the errors per API call.

Also, can you run metrics-helper integration tests against these changes for coverage?

[jayanthvn]

Good to add some UTs and integration tests around this.

[jdn5126]

Let's add some unit test cases in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/cmd/cni-metrics-helper/metrics/metrics_test.go and an integration test case in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/test/integration/metrics-helper/metric_helper_test.go .

Let's also add a per-API error count like awsAPIErr:

[jaydeokar]

Added unit test and integration test for this change. Also added the new metric for tracking EC2 Api failure count ec2ApiErrCount.

[jayanthvn]

There should be create tags isn't it?

[jaydeokar]

It's added to be tracked. But it looks like when I took the logs that function was probably not hit until then. Do we want to explicitly check for that scenario ?

[jayanthvn]

Nvm, we are incrementing the value.