livy ssl issue

[cocampbe]

Apache Airflow version: 2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.17.6

• OS (e.g. from /etc/os-release): oracle linux 7.9
• Kernel (e.g. uname -a): Linux airflow-worker-1 4.14.35-2047.502.4.el7uek.x86_64 Bring in more resolution to hivestats #2 SMP Mon Apr 5 13:21:44 PDT 2021 x86_64 GNU/Linux

What happened:

Got the following trying to user the livy operator. I have added the certs to the image and ran update-ca-certs. I can use the curl command from the worker pod and the response is as expected. But pyopenssl is not able to verify the cert. I've changed the actual server name.

[2021-05-12 20:40:04,826] {taskinstance.py:1455} ERROR - HTTPSConnectionPool(host='host.example.com', port=8998): Max retries exceeded with url: /batches (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 488, in wrap_socket
    cnx.do_handshake()
  File "/home/airflow/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/home/airflow/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
    _raise_current_error()
  File "/home/airflow/.local/lib/python3.8/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

here is trhe output from curl running in the pod.

airflow@airflow-worker-1:/opt/airflow$ curl -I https://host.example.com:8998/
HTTP/1.1 401 Unauthorized
WWW-Authenticate: MAPR-Negotiate
WWW-Authenticate: Basic realm="WebLogin"
Set-Cookie: hadoop.auth=; Path=/; Domain=.example.com; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Secure; HttpOnly
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 519
Server: Jetty(9.4.22.v20191022)

What you expected to happen:

No ssl error

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

You can pass extra_options' "verify" parameter and point it to "update-ca,certs" file.

[potiuk]

please let me know if it worked for you

[potiuk]

Closing it for now unless more information/verification is provided.

[cocampbe]

Sorry if this sounds ignorant. But are you referring to the livy connection?

…

On Thu, May 13, 2021, 10:55 AM Jarek Potiuk ***@***.***> wrote: You can pass extra_options' "verify" parameter and point it to "update-ca,certs" file. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#15806 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ34AJBGRR2BFJKZQ2LAFDTNPY57ANCNFSM44ZLALLA> .

[potiuk]

The operator. Look at the Livy Operator. It is based on the HttpOperator and it has 'extra_options' field. You can pass there the same kind of dictionary as for the HttpOperator/Hook. One of the values might be 'verify'. verify might be False/True or it also can be a pth to the root certificate used to verify the server side

[cocampbe]

@potiuk I am about to give this a try. Thanks for the help. I'll let you know the outcome.

[potiuk]

More info here: https://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

[cocampbe]

It worked. Just to test I set verify to False in the dictionary. I need to change it to use the ca cert, but want to see if it would work. Thanks again for your help. DM me your paypal info and I would be happy to send you some beer money. ;)

[potiuk]

Cool! No worries about beer money. Happy that it helped.

[cocampbe]

Passing it the certs directory worked.