[SSL: CERTIFICATE_VERIFY_FAILED] when accessing ESA ODP via HTTPS

[forman]

On Windows, I get the following error if ECT accesses ESA ODP service via HTTPS:

self = <urllib.request.HTTPSHandler object at 0x000001A3CE888EB8>
http_class = <class 'http.client.HTTPSConnection'>
req = <urllib.request.Request object at 0x000001A3CEED5518>
http_conn_args = {'check_hostname': None, 'context': None}
host = 'esgf-index1.ceda.ac.uk'
h = <http.client.HTTPSConnection object at 0x000001A3CEFECA20>

    def do_open(self, http_class, req, **http_conn_args):
        """Return an HTTPResponse object for the request, using http_class.

            http_class must implement the HTTPConnection API from http.client.
            """
        host = req.host
        if not host:
            raise URLError('no host given')

        # will parse host:port
        h = http_class(host, timeout=req.timeout, **http_conn_args)
        h.set_debuglevel(self._debuglevel)

        headers = dict(req.unredirected_hdrs)
        headers.update(dict((k, v) for k, v in req.headers.items()
                            if k not in headers))

        # TODO(jhylton): Should this be redesigned to handle
        # persistent connections?

        # We want to make an HTTP/1.1 request, but the addinfourl
        # class isn't prepared to deal with a persistent connection.
        # It will try to read all remaining data from the socket,
        # which will block while the server waits for the next request.
        # So make sure the connection gets closed after the (only)
        # request.
        headers["Connection"] = "close"
        headers = dict((name.title(), val) for name, val in headers.items())

        if req._tunnel_host:
            tunnel_headers = {}
            proxy_auth_hdr = "Proxy-Authorization"
            if proxy_auth_hdr in headers:
                tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
                # Proxy-Authorization should not be sent to origin
                # server.
                del headers[proxy_auth_hdr]
            h.set_tunnel(req._tunnel_host, headers=tunnel_headers)

        try:
            try:
                h.request(req.get_method(), req.selector, req.data, headers)
            except OSError as err: # timeout error
>               raise URLError(err)
E               urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)>

[forman]

Workaround: open https://esgf-index1.ceda.ac.uk/search/esacci-ceda/ in your internet browser (I used Chrome) so that a valid SSL certificate will be made available to Windows.

[forman]

Closing this as it has never occurred again.

[kbernat]

Looks like problem still exists, here is exception from AppVeyor (clean windows env.)

cate\ds\esa_cci_odp.py:136: in _fetch_solr_json
    with urllib.request.urlopen(url, timeout=timeout) as response:
C:\Miniconda35-x64\envs\cate\lib\urllib\request.py:163: in urlopen
    return opener.open(url, data, timeout)
C:\Miniconda35-x64\envs\cate\lib\urllib\request.py:466: in open
    response = self._open(req, data)
C:\Miniconda35-x64\envs\cate\lib\urllib\request.py:484: in _open
    '_open', req)
C:\Miniconda35-x64\envs\cate\lib\urllib\request.py:444: in _call_chain
    result = func(*args)
C:\Miniconda35-x64\envs\cate\lib\urllib\request.py:1297: in https_open
    context=self._context, check_hostname=self._check_hostname)

...

            except OSError as err: # timeout error
>               raise URLError(err)
E               urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)>
C:\Miniconda35-x64\envs\cate\lib\urllib\request.py:1256: URLError
============== 2 failed, 333 passed, 33 skipped in 26.00 seconds ==============
Command exited with code 1

[kbernat]

@forman
Problem still exists and it's easy to reproduce using AppVeyor build server

[forman]

It's back.

[forman]

• https://stackoverflow.com/questions/1087227/validate-ssl-certificates-with-python
• https://stackoverflow.com/questions/30405867/how-to-get-python-requests-to-trust-a-self-signed-ssl-certificate

[kbernat]

It's a problem on windows where python uses Windows's system certificate store.
@forman For v1.0 release it should be marked as a known issue.

[forman]

Agreed.

[papesci]

This issue has been fixed creating an unverified context before to access https services. Has been integrated in cate2.0-dev17