openssl 3: Can't construct RSA keys from any available serializable key data

[Esaron]

There appears to be no way to construct a key from any parameters. This has come up in several other issues, but the approaches that have been said to work don't seem to actually work.

$ ruby --version
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux-musl]

irb(main):001:0> puts OpenSSL::VERSION; puts OpenSSL::OPENSSL_VERSION
3.1.0
OpenSSL 3.1.2 1 Aug 2023
=> nil

I tried to use an approach adapted from Pushpad referenced in this issue comment (and others elsewhere), but didn't have any luck, and I'm trying to work with RSA keys specifically.

For all the below code:
foo = OpenSSL::PKey::RSA.new(2048)

Approach adapted from COSE referenced in this issue comment:

irb(main):104:0> foo.params.keys
=> ["n", "e", "d", "p", "q", "dmp1", "dmq1", "iqmp"]
irb(main):106:0> seq = [OpenSSL::ASN1::Integer.new(0)] + foo.params.values.map { OpenSSL::ASN1::Integer.new(_1.to_s(2)) }
=>
[#<OpenSSL::ASN1::Integer:0x00007fd2fc2e9b50 @indefinite_length=false, @tag=2, @tag_class=:UNIVERSAL, @tagging=nil, @value=0>,
...
irb(main):107:0> seq1 = OpenSSL::ASN1::Sequence(seq)
=>
#<OpenSSL::ASN1::Sequence:0x00007fd2fc33a500
...
irb(main):108:0> OpenSSL::PKey::RSA.new(seq1.to_der)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key: unsupported (OpenSSL::PKey::RSAError)

Even just using the BNs that exist on the just-generated key without stringifying and creating new BNs doesn't work:

irb(main):121:0> OpenSSL::PKey::RSA.new(seq1.to_der)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key: unsupported (OpenSSL::PKey::RSAError)
irb(main):122:1* seq = [
irb(main):123:1*   OpenSSL::ASN1::Integer.new(0),
irb(main):124:1*   OpenSSL::ASN1::Integer.new(foo.params['n']),
irb(main):125:1*   OpenSSL::ASN1::Integer.new(foo.params['e']),
irb(main):126:1*   OpenSSL::ASN1::Integer.new(foo.params['d']),
irb(main):127:1*   OpenSSL::ASN1::Integer.new(foo.params['p']),
irb(main):128:1*   OpenSSL::ASN1::Integer.new(foo.params['q']),
irb(main):129:1*   OpenSSL::ASN1::Integer.new(foo.params['dmp1']),
irb(main):130:1*   OpenSSL::ASN1::Integer.new(foo.params['dmq1']),
irb(main):131:1*   OpenSSL::ASN1::Integer.new(foo.params['iqmp']),
irb(main):132:0> ]
=>
[#<OpenSSL::ASN1::Integer:0x00007fd2fdddd678 @indefinite_length=false, @tag=2, @tag_class=:UNIVERSAL, @tagging=nil, @value=0>,
...
irb(main):133:0> seq1 = OpenSSL::ASN1::Sequence(seq)
=>
#<OpenSSL::ASN1::Sequence:0x00007fd2fae66908
...
irb(main):134:0> OpenSSL::PKey::RSA.new(seq1.to_der)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key: unsupported (OpenSSL::PKey::RSAError)

Constructing a public key seems fine:

irb(main):142:1* seq = [
irb(main):143:1*   OpenSSL::ASN1::Integer.new(foo.params['n']),
irb(main):144:1*   OpenSSL::ASN1::Integer.new(foo.params['e']),
irb(main):145:0> ]
=>
[#<OpenSSL::ASN1::Integer:0x00007fd315124b08
...
irb(main):146:0> seq1 = OpenSSL::ASN1::Sequence(seq)
=>
#<OpenSSL::ASN1::Sequence:0x00007fd305d2e5d0
...
irb(main):147:0> OpenSSL::PKey::RSA.new(seq1.to_der)
=> #<OpenSSL::PKey::RSA:0x00007fd3107ad2b8 oid=rsaEncryption>
irb(main):148:0> pub = _
=> #<OpenSSL::PKey::RSA:0x00007fd3107ad2b8 oid=rsaEncryption>
irb(main):149:0> pub.public?
=> true
irb(main):150:0> pub.private?
=> false
irb(main):151:0> pub.public_encrypt('foo')
=> "\"|Y\x7Fq\xE2\x00\xC3o\xB4\xF8g\xB5\xC3/\xE6\xD3r\x87*\xA7!h\x87\xD4\xA34\xA6\r\xF3\x84\x12\xF2\xCCW^\xA3\xB6\xFC\xAEO\xEC\t\xCCX9h\xF6\x87\xB2\xF0k\xA0\xCC9\xEE\x9Aq\x83*\xB6l\xBCp\xDDu\xC3\xFB\x05\xDAbT\x1A\xC2\x15\xAF\x1D\xE1\"V\x8C\xC7;y\xBF#\x17x#\xE9\xE07m\xD9I]\xCDw\xA4B\xB1\xDA\xAB,\x8E4q\xB2b\xAA3Oa\xB5\b\xD8)\xF9\x06\xBEUH\xB9\xC1\xEF@\x8A\xFB\xC4\xBA<\x91\xFA\x8F\xCE\xB8h%\x7F@{\r\xBE\xEFP\xE3G\x91\"\xF5\xFD\x9B\xE0\x84\x88\x98\"|\xEF<\xF3\x04\xBF\xDA\xC0\xB6\xFB!\xCDb\xE6\xC5\xFEF\xE5B\xFB\"\x1C\xA5\x85\e\x81\x0E\x8E\x90\x93\x8B\x93\x8F4\xF7\xA9\x99\xFB\xD9\xA7w\xD1v\xE25\xF3\\\x19>\x88wO\xAD\xF7\xCC\\'\x16\x9F?\xC1\xC2Z\xB8\xAB\xDC\x95\xB6\xED\x8C\x06\xA6\xA3[\xE9U\xF6p'\xA3j\x06vsx\xDCea:\x13\x95\x1C9!{\xB4>\x0FK"
irb(main):152:0> foo.private_decrypt(_)
=> "foo"

The der output of the key and the manually constructed sequence also match:

# Reconstructed seq1 as done in irb(main):122:1 - irb(main):133:1
irb(main):202:0> foo.to_der == seq1.to_der
=> true

If I take the serialized der output of the key and attempt to read it in in an environment using OpenSSL 1.1.1, that works and the key is intact:

# der = <foo.to_der from above>
[12] pry(main)> puts OpenSSL::OPENSSL_VERSION
OpenSSL 1.1.1n  15 Mar 2022
=> nil
[13] pry(main)> OpenSSL::PKey::RSA.new(der)
=> #<OpenSSL::PKey::RSA:0x00007fd429c11268 oid=rsaEncryption>
[14] pry(main)> foo111 = OpenSSL::PKey::RSA.new(der)
=> #<OpenSSL::PKey::RSA:0x00007fd429c341f0 oid=rsaEncryption>
[15] pry(main)> foo111.private_decrypt("\"|Y\x7Fq\xE2\x00\xC3o\xB4\xF8g\xB5\xC3/\xE6\xD3r\x87*\xA7!h\x87\xD4\xA34\xA6\r\xF3\x84\x12\xF2\xCCW^\xA3\xB6\xFC\xAEO\xEC\t\xCCX9h\xF6\x87\xB2\xF0k\xA0\xCC9\xEE\x9Aq\x83*\xB6l\xBCp\xDDu\xC3\xFB\x05\xDAbT\x1A\xC2\x15\xAF\x1D\xE1\"V\x8C\xC7;y\xBF#\x17x#\xE9\xE07m\xD9I]\xCDw\xA4B\xB1\xDA\xAB,\x8E4q\xB2b\xAA3Oa\xB5\b\xD8)\xF9\x06\xBEUH\xB9\xC1\xEF@\x8A\xFB\xC4\xBA<\x91\xFA\x8F\xCE\xB8h%\x7F@{\r\xBE\xEFP\xE3G\x91\"\xF5\xFD\x9B\xE0\x84\x88\x98\"|\xEF<\xF3\x04\xBF\xDA\xC0\xB6\xFB!\xCDb\xE6\xC5\xFEF\xE5B\xFB\"\x1C\xA5\x85\e\x81\x0E\x8E\x90\x93\x8B\x93\x8F4\xF7\xA9\x99\xFB\xD9\xA7w\xD1v\xE25\xF3\\\x19>\x88wO\xAD\xF7\xCC\\'\x16\x9F?\xC1\xC2Z\xB8\xAB\xDC\x95\xB6\xED\x8C\x06\xA6\xA3[\xE9U\xF6p'\xA3j\x06vsx\xDCea:\x13\x95\x1C9!{\xB4>\x0FK")
=> "foo"

These are some other simpler things I tried, just for reference:

irb(main):013:0> OpenSSL::PKey::RSA.new(foo)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key: unsupported (OpenSSL::PKey::RSAError)
irb(main):014:0> OpenSSL::PKey::RSA.new(foo.to_der)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key: unsupported (OpenSSL::PKey::RSAError)
irb(main):015:0> OpenSSL::PKey::RSA.new(foo.to_pem)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key (OpenSSL::PKey::RSAError)
irb(main):016:0> OpenSSL::PKey.read(foo.to_pem)
(irb):16:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
irb(main):017:0> OpenSSL::PKey.read(foo.to_der)
(irb):17:in `read': Could not parse PKey: unsupported (OpenSSL::PKey::PKeyError)
irb(main):018:0> OpenSSL::PKey.read(asn1.to_der)
(irb):18:in `read': Could not parse PKey: unsupported (OpenSSL::PKey::PKeyError)

Encryption/decryption via the openssl cli works fine with the generated pems, but I can't find any mechanism using any of the serializable data to instantiate a private key to do anything in Ruby. Is there a good reason why #555 can't be merged if it's working? Do you see any problems with how I've implemented the solutions that have been proposed before, or have the required approaches changed?

[Esaron]

It's probably important to note that I am running in FIPS mode. I wonder if #615 will resolve this issue? If so, and if #651 results in a new release in the near future, this issue will also be resolved.

[rhenium]

What is foo? Is it an instance of OpenSSL::PKey::RSA?

irb(main):014:0> OpenSSL::PKey::RSA.new(foo.to_der)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key: unsupported (OpenSSL::PKey::RSAError)

This is weird.

[Esaron]

I apologize, I failed to include some code. I will add it back. foo is just foo = OpenSSL::PKey::RSA.new(2048)

IE -

irb(main):003:0> OpenSSL::PKey::RSA.new(OpenSSL::PKey::RSA.new(2048).to_der)
/usr/local/lib/ruby/3.2.0/openssl/pkey.rb:356:in `initialize': Neither PUB key nor PRIV key: unsupported (OpenSSL::PKey::RSAError)

[Esaron]

I did build the gem from master today and there are no longer issues building keys in the same environment, so no new code changes appear to be required and as soon as there's a new release this should be resolved.