[Python] Latest update breaks https/ssl

[blueflame2020]

For new Package Requests, see the guidelines

Setup

Package Name: Python/SickBeard Custom
Package Version: 2.7.18-23/20200804-8

NAS Model: DS415+
NAS Architecture: Intel Atom C2538 (Avoton)
DSM version: DSM 6.2.3-25426 Update 2

Expected behavior

After upgrade to latest python version:
Sickbeard shuold work like previous python version (2.7.14-19)
Sickbeard should open when https is enabled and browsing to https://<NAS_IP>:8083
Sickbeard should be able to connect to tvdb and venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt without

URL error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)

errors.

Actual behavior

SickBeard web page does not show.
Chrome reports "The connection was reset." when going directly to https://<NAS_IP>:8083
NGINX proxy server shows:

2020/08/10 11:18:37 [error] 13031#13031: *183858 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: <CLIENT_IP>, server: sickbeard.xxxx.nl, request: "GET /favicon.ico HTTP/1.1", upstream: "https://<NAS_IP>:8083/favicon.ico", host: "sickbeard.xxxx.nl", referrer: "https://sickbeard.xxxx.nl/home/"

Sickbeard does run. (service is started and running and the logs show startup and backlog entries indicating processing of files etc. is continuing.)
On stopping SickBeard via package center, disabling https in the config.ini file and starting SickBeard again, the webpage is reachable at http://NAS_IP>:8083 (but in plain text, without certificates).
This is the case with both SickBeard self signed certificates as with own valid certificates.
Sickbeard can't search new series because connection to tvdb fails with ssl error.

Steps to reproduce

1. Previous versions of python and latest version of sickbeard installed and configured with own valid certificate or self signed certificate (https enabled)
2. Update Python package via package center
3. login to sickbeard and search for a show.

Package log

Check Package Center or /usr/local/{package}/var/
Sickbeard install log:

Mon Aug 10 08:56:25 CEST 2020
===> Step preuninst. USER=sickbeard-custom GROUP=sc-download SHARE_PATH=
Removing service configuration sickbeard-custom.sc
Mon Aug 10 08:56:26 CEST 2020
===> Step postuninst. USER=sickbeard-custom GROUP=sc-download SHARE_PATH=
Removing user sc-sickbeard-custom
Mon Aug 10 09:03:44 CEST 2020
===> Step preinst. USER=sickbeard-custom GROUP=sc-download SHARE_PATH=
Invoke service_preinst
Mon Aug 10 09:03:51 CEST 2020
===> Step postinst. USER=sickbeard-custom GROUP=sc-download SHARE_PATH=
Installing service configuration /var/packages/sickbeard-custom/conf/sickbeard-custom.sc
Adding 'sc-sickbeard-custom' to 'sc-download'
Group Name: [sc-download]
Group Type: [AUTH_LOCAL]
Group ID:   [65537]
Group Members:
0:[SB]
1:[sc-sickbeard-custom]
Invoke service_postinst
New python executable in /volume1/@appstore/sickbeard-custom/env/bin/python
Installing setuptools, pip, wheel...
done.

Other logs

No log entries in Logs/cherrypy.log after enabling weblogs via config.ini and https is enabled.
Log entry in Logs/sickbeard.log

2020-08-15 13:29:53 INFO     CHECKVERSION :: Check scene exceptions update
2020-08-15 13:29:54 WARNING  CHECKVERSION :: URL error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727) while loading URL http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt
2020-08-15 13:29:54 ERROR    CHECKVERSION :: Check scene exceptions update failed. Unable to get URL: http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt

and

2020-08-10 14:41:16 DEBUG    CP Server Thread-5 :: Trying to find Show on thetvdb.com with: Star Trek Strange New Worlds
2020-08-10 14:41:16 DEBUG    CP Server Thread-5 :: Searching for Show with searchterm: 'Star Trek Strange New Worlds' on URL http://thetvdb.com/api/GetSeries.php?seriesname=Star+Trek+Strange+New+Worlds&language=en
2020-08-10 14:41:16 WARNING  CP Server Thread-5 :: URL error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727) while loading URL http://thetvdb.com/api/GetSeries.php?seriesname=Star+Trek+Strange+New+Worlds&language=en
2020-08-10 14:41:16 ERROR    CP Server Thread-5 :: Unable to get URL: http://thetvdb.com/api/GetSeries.php?seriesname=Star+Trek+Strange+New+Worlds&language=en

[ymartin59]

@hgy59 May you please help about it.

[Hamuko]

Getting a similar error except on the Python3 package. Not actually with Sickbeard, but rather with a different Python-dependent piece of software.

<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)>

Python 3.7.7-12, DS920+, DSM 6.2.3-25426 Update 2.

[kukoarmas]

Same problem accessing AWS S3 with python3 boto package. I read it could happen with older versions of boto because it lacked SNI support, but I also tried with latest version. So if it's a generic problem it could be related with the CA bundle file packaged with some python package

[BenjV]

Has nothing to do with Python but with the applications written in python and website who do a changeover from optional to mandatory ssl verification.
If the application does not include a client certificate in the call to those website, it will fail instead of just warn.

In the past on a lot of websites this verification was not mandatory.

[blueflame2020]

Hi BenjV,

Thanks for your reply. Could you then please explain why it starts working again with the previous python package installed?

[BenjV]

That is because in older versions of python the ssl verification was not mandatory and ssl errors were just ignored
The python developers rightly so changed that in the latest release.
Ignoring an ssl verification failure is of course a major security risk.

So the problem is two fold, the website are changing and now reply with ssl errors if detected and python is not longer ignoring those errors.
The application should do a ssl verification with a correct certificate,.
Lots of applications don't do that correctly and access "https" websites exactly the same way as "http" websites but they should add a client certificate when accessing a 'https" website.

In other words that applicatie is a security risk and not compatible with the latest python version.

But this is still not an SynoCommunity issue, the Python 2 package just installs the latest Python version.

[kukoarmas]

To "fix" the problem you need to specify a ca bundle file containing the CA that issued the server cert you are trying to connect.
For example, I'm using boto to send backups to S3, and I had to setup s3cmd to use the central CA bundle file with the following configuration in .s3cfg file:
ca_certs_file = /etc/ssl/certs/ca-certificates.crt

[BenjV]

This has nothing to do with this issue.

The log file states that the application is trying to use "http" and both sites are "https only" at this moment.
So it is not an issue of missing certificate but of a wrong call to those websites.

By the way, Sickbeard is a really old fork and is no longer maintained for years now.
I would suggest to remove it and replace it with SickChill.
You can install it with the Package "Sickbear Custom"

[Safihre]

@BenjV I think in this case the problem is that the certificates are not being correctly picked up in the new packages..
The change to SSL verification in Python-core has been implemented in Python 2 and 3 already a few versions ago, so this shouldn't suddenly become a problem.

[BenjV]

Hmm ...
I did some test on my test Nas, my production Nas and Windows PC which all have the latest python 2 installed and could not reproduce the problem.

I used the Url from the error message posted by @blueflame2020 :

import urllib
print urllib.urlopen("http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt").read()

Then I checked the python version and got this:

Python 2.7.12 (default, May 12 2020, 04:48:57)
[GCC 4.9.3 20150311 (prerelease)] on linux2

While the package center tells me I have python 2.7.18-23

[Safihre]

Could you try with a httpS URL? And indeed with the right package :)

[BenjV]

I stand corrected, I started the wrong python which is included with DSM.
This is the result with the correct python version 2.7.18

print urllib.urlopen("http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt").read()
Traceback (most recent call last):
File "", line 1, in
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 87, in urlopen
return opener.open(url)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 215, in open
return getattr(self, name)(url)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 366, in open_http
return self.http_error(url, fp, errcode, errmsg, headers)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 379, in http_error
result = method(url, fp, errcode, errmsg, headers)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 675, in http_error_301
return self.http_error_302(url, fp, errcode, errmsg, headers, data)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 644, in http_error_302
headers, data)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 671, in redirect_internal
return self.open(newurl)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 215, in open
return getattr(self, name)(url)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 445, in open_https
h.endheaders(data)
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 1078, in endheaders
self._send_output(message_body)
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 894, in _send_output
self.send(msg)
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 856, in send
self.connect()
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 1303, in connect
server_hostname=server_hostname)
File "/volume1/@appstore/python/lib/python2.7/ssl.py", line 369, in wrap_socket
_context=self)
File "/volume1/@appstore/python/lib/python2.7/ssl.py", line 599, in init
self.do_handshake()
File "/volume1/@appstore/python/lib/python2.7/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)

and for https:

print urllib.urlopen("https://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt").read()
Traceback (most recent call last):
File "", line 1, in
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 87, in urlopen
return opener.open(url)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 215, in open
return getattr(self, name)(url)
File "/volume1/@appstore/python/lib/python2.7/urllib.py", line 445, in open_https
h.endheaders(data)
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 1078, in endheaders
self._send_output(message_body)
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 894, in _send_output
self.send(msg)
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 856, in send
self.connect()
File "/volume1/@appstore/python/lib/python2.7/httplib.py", line 1303, in connect
server_hostname=server_hostname)
File "/volume1/@appstore/python/lib/python2.7/ssl.py", line 369, in wrap_socket
_context=self)
File "/volume1/@appstore/python/lib/python2.7/ssl.py", line 599, in init
self.do_handshake()
File "/volume1/@appstore/python/lib/python2.7/ssl.py", line 828, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)

[BenjV]

If I use requests with a certificate file from certifi all is working OK.
This code works fine for both http and https

import certifi, requests
print requests.get("http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt", verify=certifi.where()).content

And surprise surprise this also works because requests because requests automatically uses certifi

import certifi, requests
print requests.get("http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt").content

So conclusion:
urllib.urlopen() does not work anymore with websites who are using sssl verification.
most likely because it looks in the wrong place for a certificate on DSM.

[Hamuko]

Can confirm the same happening on the Python 3 version.

Python 3.7.7 (default, Aug 18 2020, 20:42:48)
[GCC 4.9.3 20150311 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> response = requests.get("http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt")
>>> response.status_code
200
>>> import urllib.request
>>> response = urllib.request.urlopen("http://venomous.github.io/sb_tvdb_scene_exceptions/exceptions.txt")
Traceback (most recent call last):
  ...
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  ...
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)>

[BenjV]

If you start python on the commandline and give the following commands you see a list where ssl is looking for a certificate

import ssl
print ssl.get_default_verify_paths()

So I have two solutions and a one workaround for this issue.
Solution 1:
Change the python packages and add a certificate file "cert.pem"to the package location (/var/packages/python/target/cert.pem)

Solution 2:
Change the Python package an create a simlink to the certificate of the certifi site-package.
So add this command to the installation script:

ls -s /var/packages/python/target/lib/python2.7/site-packages/certifi/cacert.pem /var/packages/python/target/cert.pem

Workaround:
Create an environment variable to suppress the ssl verification just like the older python package did:

export PYTHONHTTPSVERIFY=0

My preference would be solution 2, which could also be used as a workaround for users as long as the package is not changed.

[Safihre]

I think the previous packages used solution 1 and used the general DSM certificate store. I remember inspecting this long ago and it sort of worked out of the box that it looked in the right location. But can't find much of a reference to it now.

[th0ma7]

Other ideas...
Solution 2b) what if the symlink was pointing towards DSM default /etc/ssl/certs/
Solution 3) From the previous PR from @hgy59 he did the following (ca0f280) for curl:

CONFIGURE_ARGS += --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt

Solution 4) Use certifi (ref: https://pypi.org/project/certifi/)
Workaround B) export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

More info on this:

• Perhaps another solution/workaround got merged: Add support for custom certificate authority and client certificates python-poetry/poetry#1325
• Perhaps not: Allow root CA bundle to be configured python-poetry/poetry#1012

[BenjV]

@th0ma7
Your Solution 4 is my Solution 2

The certifi package is included in the Python package as site-package so it is already present and a simlink to the cacert.pem of that certifi package is in my opinion the best solution.

That way everything is included in the Python package and not dependent on system certificates, which could change.

[blueflame2020]

The previous version (Python 2.7.14 (default, May 28 2018, 14:47:18)) where everything worked has this:

DefaultVerifyPaths(cafile='/var/packages/python/target/cert.pem', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/var/packages/python/target/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/var/packages/python/target/certs')

ls -al /var/packages/python/target/cert.pem
-rw-r--r--  1 root root 215556 May 28  2018 cert.pem

And if I understand you correctly, the new version lacks this file right?

[BenjV]

Correct, the latest package does not include a certificate in that location.

I did some investigation and the reason that normally a python release does and should not distribute a certificate is that certificates expires.
So if a certificate is part of a python package then you are forced to reinstall new versions of such a package periodically when the certificate expires.

Actually an application that uses ssl should include a certificate and keep it up-to-date.
My own python application uses the certifi site-package and download a newer version is it become available on the certifi github.

[BenjV]

I have another solution.
Create a scheduled task to run for example once a month to get the latest certificate from the certifi github
This way you can keep the certificate up to date
Put this in that task:

wget -O /var/packages/python/target/cert.pem 'https://raw.githubusercontent.com/certifi/python-certifi/master/certifi/cacert.pem'

Make sure that the task is run as root!!!!

[th0ma7]

@BenjV I would actually use both... so at install time you do have an updated cacert.pem then on a ... monthly? quarterly? basis your file gets updated.

[BenjV]

Agree, but until the python package is changed, everybody could use the wget command to get everything working for now.

[blueflame2020]

Copying the file cert.pem back after update (and chmod 644 the file) does resolve the SSL: CERTIFICATE_VERIFY_FAILED errors, so thank you for that @BenjV !

Unfortunately the other issue is not resolved with this. (sickbeard unable to run in https mode with self signed or own generated certificates also not after adding the CA and intermediate certs to the server.crt file for sickbeard (cert first, then intermediate, then root) and the CA cert to the certs.pem file.)

@th0ma7 I tried your workaround B before opening this issue but I could not get this to work. (added it to the startup script of sickbeard at /var/packages/sickbeard-custom/scripts/service-setup)

P.s. the script should look like:

wget -O /var/packages/python/target/cert.pem 'https://raw.githubusercontent.com/certifi/python-certifi/master/certifi/cacert.pem'

p.s.2: the fork that I use is kinda still maintained: https://github.com/junalmeida/Sick-Beard
But I will look into SickChill

[BenjV]

Sickbeard is just not made for https access.
Accessing local application via https seems useless to me anyway.

And the wget command I supplied was already the correct one, yours is simply an alias.

[blueflame2020]

If you put yours in a script, the file will also contain all github http stuff ;)

And lets agree to disagree on https for local applications ;)

[BenjV]

No I tested mine and it just pulls only the one file so you probably made a mistake.
The command specifies the file so it is impossble that it pulls more.

[blueflame2020]

@BenjV

 wget -O /var/packages/python/target/cert2.pem 'https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem'
--2020-08-26 13:54:32--  https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem
Resolving github.com... 140.82.121.4
Connecting to github.com|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/var/packages/python/target/cert2.pem'

/var/packages/python/target/cert2.pem                    [ <=>                                                                                                                   ]   1.16M  7.18MB/s    in 0.2s

2020-08-26 13:54:33 (7.18 MB/s) - '/var/packages/python/target/cert2.pem' saved [1214077]

Content of file (first ten lines):

head /var/packages/python/target/cert2.pem





<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://github.githubassets.com">

So, what am I doing wrong here?

[BenjV]

Apparently I did something wrong.
I tried it again with my link but now I got also a html file just like you.

So your link is the correct one and I have change mine.

Sorry.

[ymartin59]

Package updates have been published with fix #4192. Please confirm and close if relevant

[groypalo]

Package updates have been published with fix #4192. Please confirm and close if relevant

Not trying to be obnoxious but what exactly do you want us to check? Synology package showing no update available for Home Assistant from SynCommunity. Or manual upgrade through CLI?

Thanks in advance for clarifying!

[hgy59]

Not trying to be obnoxious but what exactly do you want us to check? Synology package showing no update available for Home Assistant from SynCommunity. Or manual upgrade through CLI?

There is a new Python2 package available (Version 2.7.18-24) in the synocommunity package center.
You can manually download and install the package while it is not updated in the package center view of your diskstation.

This should fix #4177 (related to home assistant).

[groypalo]

Can this fix be included in the Python3 package? I only had that one installed for Home Assistant. I tried installin the one you refer above but does not fix the issue.

[ymartin59]

hg59 made a mistake. You have to grab Python 3 package update

[groypalo]

hg59 made a mistake. You have to grab Python 3 package update

Same error on my end

OpenSSL> s_client -crlf -connect api.sense.com:443 -servername www.sense.com -showcerts
CONNECTED(00000003)
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify error:num=19:self signed certificate in certificate chain

[BenjV]

This has nothing to do with the issue of this topic.

You have a problem in your own environment, most likely you have put a self signed certificate in the system certificate path.

You can find that path by issueing this command on the Openssl prompt:

version -d

[ymartin59]

I consider it as fixed.