Rate limit exceeded when loading ALB rules

[alexwen]

It Appears as though the ALB rules checking quickly exceeds the number of requests allowed for allowed for the endpoint. We have 150+ ALBs currently.

Version

0.11.0

Installation Method

Through virtualenv

Supporting Software Versions

Python 2.7.10
Virtualenv 15.1.0

Actual Output

Traceback (most recent call last):
File "bin/aws-limit-checker.py", line 186, in
main()
File "bin/aws-limit-checker.py", line 157, in main
c.find_usage()
File "/home/jenkins/shiningpanda/jobs/b0428645/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/awslimitchecker/checker.py", line 342, in find_usage
cls.find_usage()
File "/home/jenkins/shiningpanda/jobs/b0428645/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/awslimitchecker/services/elb.py", line 71, in find_usage
alb_usage = self._find_usage_elbv2()
File "/home/jenkins/shiningpanda/jobs/b0428645/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/awslimitchecker/services/elb.py", line 140, in _find_usage_elbv2
lb['LoadBalancerName']
File "/home/jenkins/shiningpanda/jobs/b0428645/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/awslimitchecker/services/elb.py", line 171, in _update_usage_for_elbv2
alc_marker_param='Marker'
File "/home/jenkins/shiningpanda/jobs/b0428645/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/awslimitchecker/utils.py", line 144, in paginate_dict
result = function_ref(*argv, **pass_kwargs)
File "/home/jenkins/shiningpanda/jobs/b0428645/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/jenkins/shiningpanda/jobs/b0428645/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/botocore/client.py", line 599, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (Throttling) when calling the DescribeRules operation (reached max retries: 4): Rate exceeded

[jantman]

@alexwen sorry to hear that. Can I ask what versions of boto3 and botocore you have installed in the virtualenv (pip freeze | grep boto)? Retries for exceptions like this should be built in to botocore itself. If I'm not mistaken, it looks like the traceback above is for an older botocore version?

If that's not the problem, I'm not sure how quickly I'll be able to get a fix out for this... the only workaround I can think of in the meantime is either retrying the whole run, or skipping the ELB service with --skip-service ELB.

[alexwen]

Hi @jantman,

Its no problem, here are the boto versions we are running, they appear to be latest:

  Downloading boto-2.48.0-py2.py3-none-any.whl (1.4MB)
  Downloading boto3-1.4.5-py2.py3-none-any.whl (128kB)
  Downloading botocore-1.5.94-py2.py3-none-any.whl (3.6MB)

I will disable ELB limits in the mean time.

Thanks for your time!

[jantman]

Yeah, those are the current releases. Thank you for reporting this and getting back to me with the versions; I'll make sure this is addressed in the next release.

[jantman]

@alexwen Apologies for how long this has sat, but I'm finally getting around to working on the next release.

I'm trying to figure out what the best way to handle this is. I can increase the maximum number of retries for the API calls, but AFAIK the ELB API rate limiting is based on a "token bucket" scheme, so I'm not sure how much this will do; I can just hope that the exponential backoff will be enough to help. I should have a branch that you could try in a little while.

[jantman]

@alexwen ok, my issues/293 branch increases the maximum number of attempts from the default of 4 to 12. With the exponential backoff, I'd think that should help. Any chance you could give that a try? The largest account that I have access to only has a handful of ALBs.

[jantman]

@alexwen I'm targeting the 1.0.0 release of awslimitchecker sometime in the next 4-5 days. If you have a chance to test this before then, great! If not, I'll merge it and call this issue closed, and we can reopen if my solution isn't sufficient.

[alexwen]

Hey @jantman thanks for looking into this I will attempt to test with your fix early this week.

[jantman]

@alexwen I'm tentatively looking to cut the 1.0.0 release in the next day or so. Any feedback you can provide on this would be greatly appreciated, though my plan is to merge anyway and worst-case the fix is insufficient and we can reopen this issue and tackle it again in a patch release.

[alexwen]

Hi @jantman - I was finally able to verify that we are no longer hitting the API rate limit. Thanks for your help on this!

[jantman]

Sure! I cut a release today to fix another bug... I'll try to get this out some time in the next week.