Key serialisation

[public]

This is a brain dump of my research of key serialisation and a space to discuss APIs for it.

Loosely there are two types of serialisation format.

1. Ones that tell you about what they contain.
2. Ones that don't.

In category 1 there are

• PEM format PKCS#1
• All flavours of PKCS#8
• OpenSSH (PEM PKCS#1 with different headers.)
• JWK

In category 2 there is

• DER format PKCS#1

I propose we ignore category 2 until someone comes up with an obvious use-case of protocol that needs it. (paramiko/conch might be this?)

Category 1 keys are not all specified for the same range of algorithms, but they all support at the very least RSA, DSA, and ECDSA. There are also RFCs in various states of dereliction that describe serialisation for other asymmetric algorithms.

Category 1 can be further broken down into (1.A) formats that support public and private keys, (1.B) formats that only support private keys and (1.C) formats that only support public keys.

In category 1.A there are

• PKCS#8
• JWK

In category 1.B there is

• PKCS#1

In category 1.C there are

• OpenSSH
• PEM formatted subjectPublicKey

PKCS#1 public keys are sometimes called "X.509 certificates" which is also kind of wrong as it's often just the raw subjectPublicKey data that is defined across various RFCs for each algorithm. I guess they called it asymmetric for a reason.

I propose we ignore serialised public keys at first, but keep them in mind...

This leaves private key loaders for

• PEM PKCS#1
• PKCS#8
• JWK

I think PEM PKCS#1 is probably the most common of these, and it also happens to involve a lot of the work needed for OpenSSH keys. OpenSSL sometimes call this the "traditional" format.

PEM PKCS#1 has 2 flavours. Encrypted, and not encrypted. Encrypted keys are described in RFC 1423 among other places. Basically they are the same as unencrypted ones but the DER is encrypted first and some additional DEK-Info headers describing the encryption parameters are added to the PEM.

PEM PKCS#1 keys do not contain the key type within their base64 encoded payload. It only exists in the PEM headers.

Also some of these formats contain additional metadata about the key that it is important to retain access too. OpenSSH contains a comment value, PKCS #8 has various key usage attributes.

Currently I imagine an interface a bit like this.

key = PKCS1PrivateKey.load_pem(data, password, backend)
key.private_key # The underlying {RSA,DSA,ECDSA....}PrivateKey

key = PKCS1PrivateKey(key.private_key, backend)
pem_data = key.dump_pem(algorithm, mode, PBKDF2HMAC(...), password)

Where unencrypted keys are created by setting basically everything to None, because you probably shouldn't be using those anyway. Additional key usage attributes and so on can easily be added to the specific formats interface.

• OpenSSL PEM private key loading
• OpenSSL PEM public key loading
• OpenSSL PEM private key saving
• OpenSSL PEM public key saving
• PKCS8 PEM private key loading
• PKCS8 PEM public key loading
• PKCS8 PEM private key saving
• PKCS8 PEM public key saving
• OpenSSH private key loading
• OpenSSH public key loading
• OpenSSH private key saving
• OpenSSH public key saving

[public]

As touched upon above there are some naming complexities to address too. For example PKCS#1 only discusses things in terms of RSA. No other key formats. Does it make sense for a PKCS1PrivateKey to ever be anything but RSA then?

There appears to be some contention as to whether the "PKCS#1" OpenSSL "BEGIN {RSA,DSA,...} {PUBLIC,PRIVATE} KEY" PEM files are even a standard format. So perhaps it should be renamed "OpenSSLPrivateKey"?

[public]

Also additional Google and GitHub code search poking suggests that maybe PKCS #8 is actually more common? Anyone have any preferences?

[reaperhulk]

PKCS8 is pretty common these days (latest OpenSSL does create the key in that format if you use openssl req -new). But PKCS1 is also extremely common and very important for a variety of applications that link against OpenSSL.

What do you see as the benefit of having the key object in this example

key = PKCS1PrivateKey.load_pem(data, password, backend)
key.private_key # The underlying {RSA,DSA,ECDSA....}PrivateKey

rather than just having them be class methods that return the data you need?

Another approach would be putting them directly on the Private/Public key objects:

private_key = RSAPrivateKey.load_pkcs1_pem(data, password, backend)
pem_data = private_key.dump_pem(algorithm, key_size, mode, kdf, password, backend)

Code duplication could become an issue with this, but at first glance it feels more natural.

[public]

Having the intermediate object gives a places for key attributes and other format specific metadata to live. e.g. OpenSSHPublicKey is going to need some kind of label field, PKCS8PrivateKey is probably going to want to expose some kind of attributes field at some point, JWK is going to want some sort of key_operations field and so on.

It also means we don't have huge numbers of member functions on each key type, and that you we don't need to worry so much about what happens when you pass data containing e.g. a DSA key to RSAPrivateKey.load_pkcs1_pem.

[public]

I'm starting to think that we should limit the complexity of the "dump_pem" serialisation end of the API. Instead of letting users pick the cipher, mode, number of PBKDF2 iterations etc, we should just pick something that is reasonably sane and widely supported by consumers of the format in question.

This would probably mean making sure that the backend was using a sensible number of PBKDF2 iterations, a good salt, a not-awful cipher etc still but it would avoid the having to expose lots of not particularly useful complexity to the user.

The only down side I can think of this is for generating keys to be loaded into particularly old software. e.g. I expect somewhere out there someone wants to load keys that encrypted with 3DES for some reason.

[public]

JWK support would be useful for any Python code hoping to integrate with Dockers forthcoming https://github.com/docker/libtrust security layer.

[reaperhulk]

I'm going to close this in favor of more specific issues (e.g. support more encryption types, OpenSSH public key serialization, etc)