Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3 error: Short octet stream on tag decoding #185

Open
williamcroberts opened this issue Nov 26, 2019 · 3 comments
Open

python3 error: Short octet stream on tag decoding #185

williamcroberts opened this issue Nov 26, 2019 · 3 comments

Comments

@williamcroberts
Copy link

williamcroberts commented Nov 26, 2019

I'm seeing an issue in cert decoding on python3 that I am not seeing on python2:

Traceback (most recent call last):
  File "./test.py", line 8, in <module>
    cert = decoder.decode(substrate, asn1Spec=rfc2459.Certificate())
  File "/home/wcrobert/.local/lib/python3.6/site-packages/pyasn1/codec/ber/decoder.py", line 1338, in __call__
    'Short octet stream on tag decoding'
pyasn1.error.SubstrateUnderrunError: Short octet stream on tag decoding

Which I can reproduce with this certificate:

-----BEGIN CERTIFICATE-----
MIIBETCBuAIJAJ0W0tvyDooPMAoGCCqGSM49BAMCMBExDzANBgNVBAMMBm15IGtl
eTAeFw0xOTExMjYxNzM0MTRaFw0yMDExMjUxNzM0MTRaMBExDzANBgNVBAMMBm15
IGtleTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABOTFZ0YJOAb39qkUJYIQxqM8
TW3fMsokFnc4oR7221+ysTS6xBHkvLUB2Xh8OVZOsCIRsZMvSrpBh7TirjIqs2Iw
CgYIKoZIzj0EAwIDSAAwRQIgRPLeuw00u5+PJx+v531MThBhBtryeLAV7s6KoeTX
hpQCIQCyyy9swRJgzBB1Op9A5KJrwMWeFwW9w1L890ub7zkGMQ==
-----END CERTIFICATE-----
from pyasn1_modules import pem, rfc2459
from pyasn1.codec.der import decoder

substrate = pem.readPemFromFile(open("cert.pem", "rb"))
cert = decoder.decode(substrate, asn1Spec=rfc2459.Certificate())

OpenSSL seems to be fine with the cert:

openssl x509 -in cert.pem -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            9d:16:d2:db:f2:0e:8a:0f
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = my key
<snip>

As well as various SSL cert checker websites, like https://www.sslchecker.com/certdecoder

I've tried a few different versions of python3 (3.5.2 and 3.6.8) and can reproduce with both. Ive also tried different versions of pyasn1 and pyasn1_modules with no luck.

williamcroberts pushed a commit to williamcroberts/tpm2-pkcs11 that referenced this issue Nov 26, 2019
Rather than hold up all forward progress on ASN1 Python3 issues,
disable Python3 testing and file a bug to re-enable it.

PyASN1 bug:
etingof/pyasn1#185

tpm2-pkcs11 bug:
tpm2-software#327

Signed-off-by: William Roberts <william.c.roberts@intel.com>
williamcroberts pushed a commit to tpm2-software/tpm2-pkcs11 that referenced this issue Nov 26, 2019
Rather than hold up all forward progress on ASN1 Python3 issues,
disable Python3 testing and file a bug to re-enable it.

PyASN1 bug:
etingof/pyasn1#185

tpm2-pkcs11 bug:
#327

Signed-off-by: William Roberts <william.c.roberts@intel.com>
@williamcroberts
Copy link
Author

In it's in the code that returns the substrate is returning an empty string.

def readPemBlocksFromFile(fileObj, *markers):
    startMarkers = dict(map(lambda x: (x[1], x[0]),
                            enumerate(map(lambda y: y[0], markers))))
    stopMarkers = dict(map(lambda x: (x[1], x[0]),
                           enumerate(map(lambda y: y[1], markers))))
    idx = -1
    substrate = ''
    certLines = []
    state = stSpam
    while True:
        certLine = fileObj.readline()

The line:

certLine = fileObj.readline()

never picks up the -----BEGIN CERTIFICATE----- scissor line because the mode flags on the file are 'rb' and readline() returns binary data. This seems very brittle, we probably want to ensure that the mode flags wouldn't cause these issues. Maybe call encode() or str() on the returned data from read? Not really a python guru, so not sure what the best fix for that would be.

williamcroberts pushed a commit to williamcroberts/tpm2-pkcs11 that referenced this issue Nov 26, 2019
python2 and python3 differ in behavior on reading from a file that was
opened with the binary flag. In python2 the read data is a str, in
python3 its bytes. This is posing an issue in the asn1 calls as outlined
in bug:
  - etingof/pyasn1#185

We can work around this for now, since pem files are always string,
remove the binary flag.

Releates to bug tpm2-software#327

Signed-off-by: William Roberts <william.c.roberts@intel.com>
williamcroberts pushed a commit to williamcroberts/tpm2-pkcs11 that referenced this issue Nov 26, 2019
python2 and python3 differ in behavior on reading from a file that was
opened with the binary flag. In python2 the read data is a str, in
python3 its bytes. This is posing an issue in the asn1 calls as outlined
in bug:
  - etingof/pyasn1#185

We can work around this for now, since pem files are always string,
remove the binary flag.

Releates to bug tpm2-software#327

Signed-off-by: William Roberts <william.c.roberts@intel.com>
@etingof
Copy link
Owner

etingof commented Nov 27, 2019

Interesting! Thank you for troubleshooting this issue! Error message is misleading.

I will push a patch and report back.

@williamcroberts
Copy link
Author

@etingof the other issue, is when even when we get past this the data returned later on when accessing the ASN1 sub fields of the cert is str, where we would probably want byte array.

I think a better fix would be ensuring that after we decode base64 we ensure that its a bytes and not str.... this py2to3 str/bytes stuff has been fun (not).

williamcroberts pushed a commit to williamcroberts/tpm2-pkcs11 that referenced this issue Nov 27, 2019
python2 and python3 differ in behavior on reading from a file that was
opened with the binary flag. In python2 the read data is a str, in
python3 its bytes. This is posing an issue in the asn1 calls as outlined
in bug:
  - etingof/pyasn1#185

We can work around this for now, since pem files are always string,
remove the binary flag and open pem w/o binary mode flag

Additionally, python3 and python 2 differe in handling str and bytes and
python3 requires more type correctness. Fix all this in a way that works
with python2 and python3

Releates to bug tpm2-software#327

Signed-off-by: William Roberts <william.c.roberts@intel.com>
williamcroberts pushed a commit to williamcroberts/tpm2-pkcs11 that referenced this issue Nov 27, 2019
python2 and python3 differ in behavior on reading from a file that was
opened with the binary flag. In python2 the read data is a str, in
python3 its bytes. This is posing an issue in the asn1 calls as outlined
in bug:
  - etingof/pyasn1#185

We can work around this for now, since pem files are always string,
remove the binary flag and open pem w/o binary mode flag

Additionally, python3 and python 2 differe in handling str and bytes and
python3 requires more type correctness. Fix all this in a way that works
with python2 and python3

Releates to bug tpm2-software#327

Signed-off-by: William Roberts <william.c.roberts@intel.com>
williamcroberts pushed a commit to tpm2-software/tpm2-pkcs11 that referenced this issue Nov 27, 2019
python2 and python3 differ in behavior on reading from a file that was
opened with the binary flag. In python2 the read data is a str, in
python3 its bytes. This is posing an issue in the asn1 calls as outlined
in bug:
  - etingof/pyasn1#185

We can work around this for now, since pem files are always string,
remove the binary flag and open pem w/o binary mode flag

Additionally, python3 and python 2 differe in handling str and bytes and
python3 requires more type correctness. Fix all this in a way that works
with python2 and python3

Releates to bug #327

Signed-off-by: William Roberts <william.c.roberts@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants