"MUPCA Root" Certificates - treated as invalid and cause error, but are walid and necessary

[MDM-79]

BPO	45312
Nosy	@pfmoore, @tjguk, @zware, @zooba, @MDM-79, @pukkandan
Dependencies	bpo-35665: Function ssl.create_default_context raises exception on Windows 10 when called with ssl.Purpose.SERVER_AUTH) attribute
Superseder	bpo-35665: Function ssl.create_default_context raises exception on Windows 10 when called with ssl.Purpose.SERVER_AUTH) attribute
Files	Untitled.png: MUPCA Root - Certificates in qestion 135081521-83d466d9-c71d-465c-96a2-652c2549c461.png: Error 1 135087546-e6fd1b05-6858-4e0f-ad3f-857c614bb15b.png: Error 2

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-28.16:11:10.264>
created_at = <Date 2021-09-28.15:49:22.455>
labels = ['expert-SSL', 'OS-windows', 'type-crash', '3.9']
title = '"MUPCA Root" Certificates - treated as invalid and cause error, but are walid and necessary'
updated_at = <Date 2021-09-28.21:57:36.256>
user = 'https://github.com/MDM-79'

bugs.python.org fields:

activity = <Date 2021-09-28.21:57:36.256>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-28.16:11:10.264>
closer = 'steve.dower'
components = ['Windows', 'SSL']
creation = <Date 2021-09-28.15:49:22.455>
creator = 'MDM-1'
dependencies = ['35665']
files = ['50312', '50313', '50314']
hgrepos = []
issue_num = 45312
keywords = []
message_count = 9.0
messages = ['402784', '402786', '402787', '402788', '402789', '402790', '402791', '402801', '402813']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'MDM-1', 'pukkandan', 'Pedjas']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = '35665'
type = 'crash'
url = 'https://bugs.python.org/issue45312'
versions = ['Python 3.9']

[MDM-79]

I just commented to the issue here https://bugs.python.org/issue35665?@ok_message=issue%2035665%20files%20edited%20ok&@template=item, but noticed "closed" so better start a new one issue, and to further update the importance of those certificates...

I came to this issue (still persistent with all python versions since 3.6) while using yt-dlp: yt-dlp/yt-dlp#1060

I obviously have the SAME problem than the guys in your link since I am from Serbia too, and those certificates "MUPCA Root" are (unfortunately-badly executed) crucial (issued by the ministry of interior - police 🙄) ones to be able too read ID cards and use personal signing certificates, and they're are all valid...
So the option to remove the faulty certificates, is a no go to me (or anyone in Serbia using their ID card - individuals, companies and entrepreneurs like me)...

Please help!

[zooba]

This needs to be a feature request against the script that you're running. They have the option of not verifying TLS certificates if they choose not to, but they are explicitly enabling the checks right now.

We can't add a command line option to disable it for them. You'll need to find the place where they work and request it there.

[MDM-79]

OK, will let the yt-dlp author pukkandan on GitHub know.
Thanks for the quick answer.

The only downside in this will be that this will have to be donne for many programs and scripts in the future; and for more and more persons (using our ID certificate will be only more preponderant as time passes).
So is it impossible to just somehow circumvent or add as exclusions those certificates? I could send them all to you..?

[zooba]

Python doesn't include any trusted certificates - it reads them from the operating system. So you'll need to get the operating system vendors to include it if you want it to be trusted by default.

Additionally, some libraries include a copy of Mozilla's bundle (usually via the certifi package) and override the operating system. You'd need them to also include it.

[pukkandan]

Hi,

I am the maintainer of the above mentioned project. I was planning to implement a patch for this. But I asked OP to report the issue here anyway since I do not believe this is the intended behavior.

For context, the issue is occurring when using the ssl.create_default_context function and not by manually adding the verify flag. For this, the default (in my opinion) should be to ignore any invalid certificates. Even the comment in the relevent code (

cpython/Lib/ssl.py

Lines 772 to 774 in 8497514

	# no explicit cafile, capath or cadata but the verify mode is
	# CERT_OPTIONAL or CERT_REQUIRED. Let's try to load default system
	# root CA certificates for the given purpose. This may fail silently.

) seem to agree with my sentiment.

I ask that you please reconsider your stance on this issue. Thanks

[zooba]

Adding Christian, as he's our expert in this area, and was also driving the other bug.

[pukkandan]

Also, the pictures uploaded by the OP are misleading since they are from a version of the code that was specifically intended for debugging the issue. the problem can be better seen in this comment yt-dlp/yt-dlp#1060 (comment)

C:\Windows\system32>py
Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> len(ssl.enum_certificates('ROOT'))
68
>>> len(ssl.enum_certificates('CA'))
39
>>> ssl.create_default_context()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\ssl.py", line 750, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 574, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 566, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: not enough data: cadata does not contain a certificate (_ssl.c:4159)
>>> exit()

[Pedjas]

This hurts number of Python applications, even those published by large players. Basically, any attempt to read any certificate (for example to load any https url) fails due to this issue.

For example:

• QGIS fails to load map tiles on https links. Python issue with certificates.

• AutoDesk Fusion 360 cannot be installed. On install, it requires online activation. Activation is done using https link. That does not work as Python fails on certificates.

And that is a bug of Python. If you check code that causes this issue you will notice problem in code.

1. When some certificate is needed Python loops and tries to load each and every certificate installed instead of loading only certificate that is actually needed and skip others.

2. No exception handling. When trying to load "bad" certificate, Python just crashes instead of graciously handle (skip) issue.

This problem occurs only with Python. No other application has such issue when handling certificates. MUPCA certificate works fine with every other application.

This issue can be easily solved with one simple if and one simple exception handler: loop through certificate. Only if certificate is the one needed try to load it. Enclose loading code within exception, if it fails, report descriptive error, and skip further. Do not allow Python to crash.

[zooba]

Looks like you should take the discussion to bpo-35665, and this one can stay closed.