Uncaught error from SES limit checks

[TimGebert]

Bug Report

SES Limit checks crash in regions where the service is not available. Similar issue to #375

First noticed the error on Friday 1st March at 12:23PM (AEDT)

If I swap to a different region (us-east-1), that supports SES, the error doesn't occur.

The error returned is a 503, Service Unavailable:
{'Error': {'Code': '503', 'Message': 'Service Unavailable'}, 'ResponseMetadata': {'HTTPStatusCode': 503, 'HTTPHeaders': {'content-length': '0', 'connection': 'keep-alive'}, 'MaxAttemptsReached': True, 'RetryAttempts': 4}}

It appears AWS might have changed their API, should the AccessDenied error that is currently caught in ses.py be changed to 503, or should a second check be added?

Version

awslimitchecker version: 6.1.4

AWS Region

ap-southeast-2 (Sydney)

Installation Method

awslimitchecker installed with command pip install awslimitchecker==6.1.4

Supporting Software Versions

Python Version: 2.7.15
Also tested on 3.7.2

Actual Output

[INFO connectable.py:90 - awslimitchecker.connectable.connect() ] Connected to ses in region ap-southeast-2
[DEBUG retry.py:210 - urllib3.util.retry.from_int() ] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
[DEBUG connectionpool.py:813 - urllib3.connectionpool._new_conn() ] Starting new HTTPS connection (1): email.ap-southeast-2.amazonaws.com:443
[DEBUG connectionpool.py:393 - urllib3.connectionpool._make_request() ] https://email.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 503 0
[DEBUG retry.py:210 - urllib3.util.retry.from_int() ] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
[DEBUG connectionpool.py:393 - urllib3.connectionpool._make_request() ] https://email.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 503 0
[DEBUG retry.py:210 - urllib3.util.retry.from_int() ] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
[DEBUG connectionpool.py:393 - urllib3.connectionpool._make_request() ] https://email.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 503 0
[DEBUG retry.py:210 - urllib3.util.retry.from_int() ] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
[DEBUG connectionpool.py:393 - urllib3.connectionpool._make_request() ] https://email.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 503 0
[DEBUG retry.py:210 - urllib3.util.retry.from_int() ] Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
[DEBUG connectionpool.py:393 - urllib3.connectionpool._make_request() ] https://email.ap-southeast-2.amazonaws.com:443 "POST / HTTP/1.1" 503 0
Traceback (most recent call last):
  File "c:\users\...\programs\python\python37-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\...\programs\python\python37-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\...\Programs\Python\Python37-32\Scripts\awslimitchecker.exe\__main__.py", line 9, in <module>
  File "c:\users\...\programs\python\python37-32\lib\site-packages\awslimitchecker\runner.py", line 415, in console_entry_point
    r.console_entry_point()
  File "c:\users\...\programs\python\python37-32\lib\site-packages\awslimitchecker\runner.py", line 409, in console_entry_point
    res = self.check_thresholds()
  File "c:\users\...\programs\python\python37-32\lib\site-packages\awslimitchecker\runner.py", line 310, in check_thresholds
    service=self.service_name)
  File "c:\users\...\programs\python\python37-32\lib\site-packages\awslimitchecker\checker.py", line 553, in check_thresholds
    cls._update_limits_from_api()
  File "c:\users\...\programs\python\python37-32\lib\site-packages\awslimitchecker\services\ses.py", line 111, in _update_limits_from_api
    resp = self.conn.get_send_quota()
  File "c:\users\...\programs\python\python37-32\lib\site-packages\botocore\client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "c:\users\...\programs\python\python37-32\lib\site-packages\botocore\client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (503) when calling the GetSendQuota operation (reached max retries: 4): Service Unavailable

Expected Output

Limit Checker handle the error gracefully.

Testing Assistance

Can help to test if required.

[bergkampsliew]

I just tried with eu-central-1 and still getting AccessDenied error.
Did you try to check with AWS Support, in case if there's issue with the infra or service in that region?

[jantman]

If I recall correctly from some previous issue, error messages for services generally change before a service is enabled in a new region, and 503s are the last step before a service goes live.

If anyone else can verify that SES in ap-southeast-2 is returning 503s, that would be great.

Given what we've seen on previous similar issues, I think we should definitely skip for both error messages. As I saw in #382, there are even cases where different accounts see different error messages for the same region...

Unfortunately I don't think I'm going to be able to get to implementing and/or releasing a fix for this for a few days, as I have some personal commitments.

@TimGebert until a fix is out, assuming you're using the CLI to run awslimitchecker, you can bypass this in the affected region with the --skip-service SES option.

[bergkampsliew]

i just tested with ap-southeast-2 and getting same error as reported.
while ap-southeast-1 returns the usual WARNING:awslimitchecker.services.ses:Skipping SES: Could not connect to the endpoint URL: "https://email.ap-southeast-1.amazonaws.com/"

[jantman]

I've been able to reproduce this myself and am working on a fix now. Should have something out soon.

[jantman]

This has been released as 6.1.5 and is now live on PyPI. Thanks so much for your help with this @TimGebert and @bergkampsliew