azure-identity | CertificateCredential fails to parse PEM certificate (Error "Failed to deserialize certificate in PEM or PKCS12 format")

[anishnair-kpmg]

• Package Name: azure-identity
• Package Version: 1.10.0
• Operating System: Windows 10
• Python Version: 3.10.4

Describe the bug
Getting 'Failed to deserialize certificate in PEM or PKCS12 format' error while using 'CertificateCredential' python module in 'azure.identity' python package. More details error logs below:

Traceback (most recent call last):
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 90, in load_pkcs12_certificate
private_key, cert, additional_certs = pkcs12.load_key_and_certificates(
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\primitives\serialization\pkcs12.py", line 147, in load_key_and_certificates
return ossl.load_key_and_certificates_from_pkcs12(data, password)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 2134, in load_key_and_certificates_from_pkcs12
pkcs12 = self.load_pkcs12(data, password)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 2151, in load_pkcs12
raise ValueError("Could not deserialize PKCS12 data")
ValueError: Could not deserialize PKCS12 data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\userfolder\Downloads\AzureCertificateBasedFileDownload.py", line 51, in
credentials = CertificateCredential(AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_CERTIFICATE_PATH)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 54, in init
client_credential = get_client_credential(certificate_path, **kwargs)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 133, in get_client_credential
cert = load_pkcs12_certificate(certificate_data, password)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 95, in load_pkcs12_certificate
six.raise_from(ValueError("Failed to deserialize certificate in PEM or PKCS12 format"), ex)
File "", line 3, in raise_from
ValueError: Failed to deserialize certificate in PEM or PKCS12 format

To Reproduce
Steps to reproduce the behavior:
Failure Scenario:
Install ‘azure-identity’ by running pip install azure-identity
Run python script that has "credentials = CertificateCredential(AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_CERTIFICATE_PATH)"

Expected Failure Message: “ValueError: Failed to deserialize certificate in PEM or PKCS12 format”

Expected behavior
To successfully read certificate

Additional context
If I uninstall latest version 'azure-identity' and specifically install azure-identity ==1.4.0 then I'm able to parse the certificate without any issues.
Additionally, I could see that issue was raised before "#21798". However, I could see this particular issue as closed. But I'm facing the same error.

[catalinaperalta]

Thanks for reaching out @anishnair-kpmg! We'll investigate asap.

[xiangyan99]

@anishnair-kpmg Thanks for reporting the issue. Could you help to check the version of cryptography library?

[anishnair-kpmg]

@xiangyan99 Sure, it's '37.0.2'. I remember trying with '3.4' as well and it was giving me same error but please let me know if you want me to try any specific cryptography version.

[xiangyan99]

Thanks for your reply. Could you help to try cryptography 3.4.8? There was a breaking change introduced in 35.0.0. Just wanted to make sure it was not the cause.

Thank you.

[anishnair-kpmg]

@xiangyan99 I have tried but getting same error. Logs below:

-----azure identity version-----
1.10.0
-----cryptography version-----
3.4.8
Azure Blob Storage v12.11.0 - Python quickstart sample
Traceback (most recent call last):
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 90, in load_pkcs12_certificate
private_key, cert, additional_certs = pkcs12.load_key_and_certificates(
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\primitives\serialization\pkcs12.py", line 28, in load_key_and_certificates
return backend.load_key_and_certificates_from_pkcs12(data, password)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 2492, in load_key_and_certificates_from_pkcs12
raise ValueError("Could not deserialize PKCS12 data")
ValueError: Could not deserialize PKCS12 data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\userfolder\Downloads\AzureCertificateBasedFileDownload-Copy.py", line 40, in
credentials = CertificateCredential(AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_CERTIFICATE_PATH)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 54, in init
client_credential = get_client_credential(certificate_path, **kwargs)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 133, in get_client_credential
cert = load_pkcs12_certificate(certificate_data, password)
File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\azure\identity_credentials\certificate.py", line 95, in load_pkcs12_certificate
six.raise_from(ValueError("Failed to deserialize certificate in PEM or PKCS12 format"), ex)
File "", line 3, in raise_from
ValueError: Failed to deserialize certificate in PEM or PKCS12 format

[xiangyan99]

Thank you for the updates.

Our loading certificate calls cryptography.hazmat.primitives.serialization.pkcs12.load_key_and_certificates

Can you try

from cryptography.hazmat.primitives.serialization import pkcs12
from cryptography.hazmat.backends.openssl.backend import backend
private_key, cert, additional_certs = pkcs12.load_key_and_certificates(
            certificate_data, password, backend=backend
        )

and see if it works?

If you only have the certificate path, you may also need

with open(certificate_path, "rb") as f:
            certificate_data = f.read()

If you don't mind, please try with the latest cryptography library (I guess the bug is in cryptography?)

Thank you.

[anishnair-kpmg]

@xiangyan99 Thank you Xiang. I have tried following code as per your suggestion:

from cryptography.hazmat.primitives.serialization import pkcs12
from cryptography.hazmat.backends.openssl.backend import backend
certificate_path  = "<full path to the certificate>"
password =None
with open(certificate_path, "rb") as f:
            certificate_data = f.read()

private_key, cert, additional_certs = pkcs12.load_key_and_certificates(
            certificate_data, password, backend=backend
        )

Please note that since password is mandatory argument, I have provided it as None since password is not required to open the certificate.

I get similar error after running the script:

Traceback (most recent call last):
  File "C:\Users\userfolder\Downloads\AzureCertificateBasedFileDownload-Copy.py", line 48, in <module>
    private_key, cert, additional_certs = pkcs12.load_key_and_certificates(
  File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\primitives\serialization\pkcs12.py", line 147, in load_key_and_certificates
    return ossl.load_key_and_certificates_from_pkcs12(data, password)
  File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 2134, in load_key_and_certificates_from_pkcs12
    pkcs12 = self.load_pkcs12(data, password)
  File "C:\Users\userfolder\AppData\Local\Programs\Python\Python310\lib\site-packages\cryptography\hazmat\backends\openssl\backend.py", line 2151, in load_pkcs12
    raise ValueError("Could not deserialize PKCS12 data")
ValueError: Could not deserialize PKCS12 data

[xiangyan99]

Thanks for your confirmation.

If your certificate is valid (I assume it is. :)), seems like there is a bug in cryptography library.

I would suggest opening issues in https://github.com/pyca/cryptography.

[ghost]

Hi @anishnair-kpmg. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[anishnair-kpmg]

/unresolve
@xiangyan99 Happy to raise the issue with pyca cryptography. However, could you please let me know then why I'm able to read the certificate in azure identity 1.4.0 version? Because I continue use the same cryptography version.

[xiangyan99]

Could you help to check the cryptography version when you use azure identity 1.4.0 and it works?

If you use the latest cryptography, could you try:

from cryptography import x509
from cryptography.hazmat.backends import default_backend
cert = x509.load_pem_x509_certificate(pem_bytes, default_backend())

[anishnair-kpmg]

@xiangyan99 I'm using latest Cryptography "37.0.2" while using azure identity "1.4.0" version. However, if I use azure identity "1.10.0" even if it is latest or older 3.4.8 cryptography version I get the same error.

Sure I will the above code which you have suggested and let you know the outcome. I assume 'pem_bytes' parameter to be passed is the content of my certificate.

[xiangyan99]

I believe a fix is merged and will be available in next release. Please feel free to reopen if it is not solved.

[ghost]

Hi @anishnair-kpmg. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @anishnair-kpmg, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

[anishnair-kpmg]

/unresolve
@xiangyan99 Thank you so much. I'm glad to hear that it's fixed and changes are merged. Could you please let me know when this is set to release?

[xiangyan99]

It would be available in July's release (which would be a beta release).

You can install the live bits if you want to try it now.

[anishnair-kpmg]

@xiangyan99 Thank you for quick response. That's helpful. Given that it's working with an older version for now, I'm happy to wait until the release. I will let you know once I validate it using latest release. Once gain thank you for fixing this issue :)

[xiangyan99]

1.11.0b2 is released with the fix.

[ghost]

Hi @anishnair-kpmg. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[mohitnmo]

Hi @anishnair-kpmg ,
I am also facing same issue, can you provide me the final solution for this!!
python version- 3.10.4 (64 bit)
azure-identity version- 1.10.0
cryptography version- 37.0.2
window 10

Thanks,
Mohit

[ghost]

Hi @anishnair-kpmg, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

[anishnair-kpmg]

@mohitnmo Try with azure-identity version '1.4.0' instead. If you are getting same error as mine, then it should work in this version. Additionally, this issue is fixed in the latest client library version as mentioned by @xiangyan99. So I would suggest to use this version as well which is in pre-release phase but available to use:
https://pypi.org/project/azure-identity/1.11.0b2/

I'm yet to try the latest release but I will soon be using this version.