IPAMD optimizations and makefile changes

[jayanthvn]

What type of PR is this?
bug

Which issue does this PR fix:
Fixes #1834
Fixes #1872

What does this PR do / Why do we need it:

1. During reconciling or even startup, IPAMD calls tryAssignCidr and adds the IPs/Prefixes to datastore once EC2 allocates the IPs and doesn't wait for IMDS to sync. In scenarios where IMDS is out of sync we will end up tagging those pods as stale during restart. Hence similar to Alloc path, we need to wait until IMDS sync is done.

2. While we wait for the IPs to be present in IMDS, we check if the requested CIDRs are available but this can be more than what the ENI supports and we end up waiting in the retry loop. Instead we need to request for min of {IPs/Prefixes needed, max supported on ENI}

3. Make file grouping.

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:
#1834 #1872

Testing done on this change:

make help

Usage:
  make <target>

Building
  all              Builds Init, CNI and metrics helper container images.
  build-linux      Build the VPC CNI plugin agent using the host's Go toolchain.
  docker           Build VPC CNI plugin & agent container image.
  docker-init      Build VPC CNI plugin Init container image.
  docker-func-test  Run the built CNI container image to use in functional testing
  multi-arch-cni-build-push  Build multi-arch VPC CNI container image.
  multi-arch-cni-init-build-push  Build VPC CNI plugin Init container image.

Run Unit Tests
  unit-test        Run unit tests
  unit-test-race   Run unit tests with race detection (can only be run natively)

Build and Run Unit Tests
  build-docker-test  Build the unit test driver container image.
  docker-unit-tests  Run unit tests inside of the testing container image.

Build metrics helper agent
  build-metrics    Build metrics helper agent.
  docker-metrics   Build metrics helper agent Docker image.

Run metrics helper Unit Tests
  metrics-unit-test  Run metrics helper unit test suite (must be run natively).
  docker-metrics-test  Run metrics helper unit test suite in a container.
  plugins          Fetch the CNI plugins

Debug script
  debug-script     Fetching debug script from awslabs/amazon-eks-ami

Formatting
  check            Run all source code checks.
  lint             Run golint on source code.
  vet              Run go vet on source code.
  docker-vet       Run go vet inside of a container.
  format           Format all Go source code files. (Note! integration_test.go has an upstream import dependency that doesn't match)

Generate ENI/IP limits
  generate-limits  Generate limit file go code

Cleanup
  clean            Clean temporary files and build artifacts from the project.

Helpers
  help             Display this help

PASS
coverage: 100.0% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry	0.002s	coverage: 100.0% of statements
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/ttime	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/ttime/mocks	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/version	[no test files]

Automation added to e2e:

Yes

Will this PR introduce any new dependencies?:

No, EC2 calls will reduce since reconciler on adding IPs would make an ec2 call now instead it will wait for IMDS sync.

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
No

Does this change require updates to the CNI daemonset config files to work?:

No

Does this PR introduce any user-facing change?:

No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[orsenthil]

LGTM.

[jayanthvn]

Thanks Senthil. I am running few UTs, will update it and the merge.

[github-actions]

This pull request is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

[jayanthvn]

/not stale. Review pending

[jayanthvn]

Ran a simple pod churn test -

1 node, replica 1 to 30 to 0 and repeated 10 runs. WARM IP = 4 and MIN IP = 10. Describe interface calls reduced quite significantly -

With 1.11.2 released image ->

With the fix ->

API Calls covering both the above timelines ->

I will be running few more tests to slow down the pod churn and measure the performance. Also will be checking the pod launch time since now we would rely on IMDS sync on increasing IP pool.

[jayanthvn]

Adding some delay between pod churns looks good too -

With 1.11.2

With fix

[jayanthvn]

This will introduce a bit of delay in the pod launch time during churn scenarios since we rely on IPs to be available in IMDS. So I will be splitting the PR, one with make file changes and reducing the describe call and another with just the dependency on IMDS for #1967 which might need more analysis.

[M00nF1sh]

/lgtm
/approve