Update Insufficient IP address logic in ipamd

[cgchinmay]

What type of PR is this?
bug

Which issue does this PR fix:
#1739

What does this PR do / Why do we need it:
Check for Insufficient CIDR blocks as well as Insufficient Free IP in subnet

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:

Testing done on this change:

Manually checked ipamd logs

{"level":"error","ts":"2021-12-02T03:34:46.067Z","caller":"ipamd/ipamd.go:934","msg":"Failed to allocate a private IP/Prefix addresses on ENI eni-01f879673fc5fd14d: InsufficientFreeAddressesInSubnet: The specified subnet does not have enough free addresses to satisfy the request.\n\tstatus code: 400, request id: aeac321e-be9f-4a13-9b2f-e6ef5985689a"}
{"level":"warn","ts":"2021-12-02T03:34:46.067Z","caller":"ipamd/ipamd.go:908","msg":"failed to allocate all available IP addresses on ENI eni-01f879673fc5fd14d, err: InsufficientFreeAddressesInSubnet: The specified subnet does not have enough free addresses to satisfy the request.\n\tstatus code: 400, request id: aeac321e-be9f-4a13-9b2f-e6ef5985689a"}
{"level":"info","ts":"2021-12-02T03:34:46.067Z","caller":"ipamd/ipamd.go:938","msg":"Trying to allocate 1 IP addresses on ENI eni-01f879673fc5fd14d"}
{"level":"debug","ts":"2021-12-02T03:34:46.067Z","caller":"ipamd/ipamd.go:938","msg":"PD enabled - false"}
{"level":"error","ts":"2021-12-02T03:34:46.420Z","caller":"ipamd/ipamd.go:938","msg":"Failed to allocate a private IP/Prefix addresses on ENI eni-01f879673fc5fd14d: InsufficientFreeAddressesInSubnet: The specified subnet does not have enough free addresses to satisfy the request.\n\tstatus code: 400, request id: 778715f3-f410-437e-9a2b-2315ad433044"}
{"level":"error","ts":"2021-12-02T03:34:46.420Z","caller":"ipamd/ipamd.go:660","msg":"failed to allocate one IP addresses on ENI eni-01f879673fc5fd14d, err: InsufficientFreeAddressesInSubnet: The specified subnet does not have enough free addresses to satisfy the request.\n\tstatus code: 400, request id: 778715f3-f410-437e-9a2b-2315ad433044: InsufficientFreeAddressesInSubnet: The specified subnet does not have enough free addresses to satisfy the request.\n\tstatus code: 400, request id: 778715f3-f410-437e-9a2b-2315ad433044"}
{"level":"debug","ts":"2021-12-02T03:34:46.420Z","caller":"ipamd/ipamd.go:795","msg":"Insufficient IP Addresses due to: InsufficientFreeAddressesInSubnet\n"}
{"level":"error","ts":"2021-12-02T03:34:46.420Z","caller":"ipamd/ipamd.go:660","msg":"Unable to attach IPs/Prefixes for the ENI, subnet doesn't seem to have enough IPs/Prefixes. Consider using new subnet or carve a reserved range using create-subnet-cidr-reservation"}
{"level":"debug","ts":"2021-12-02T03:34:51.421Z","caller":"ipamd/ipamd.go:659","msg":"IP pool is too low: available (8) < ENI target (1) * addrsPerENI (9)"}
...
...
...
{"level":"debug","ts":"2021-12-02T03:36:21.461Z","caller":"ipamd/ipamd.go:660","msg":"Starting to increase pool size"}
{"level":"debug","ts":"2021-12-02T03:36:21.461Z","caller":"ipamd/ipamd.go:660","msg":"Recently we had InsufficientCidr error hence will wait for 2m0s before retrying"}

Automation added to e2e:

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
No

Does this change require updates to the CNI daemonset config files to work?:

No

Does this PR introduce any user-facing change?:

Yes, updated ipamd logs, increase back off (2 min) for ENI allocation if the subnet doesnt have any free IPs

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jayanthvn]

Looking into it. Can you update the branch?

[jayanthvn]

Rest looks good, just minor nit.

[jayanthvn]

lgtm