macOS 11: System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestUsageTests.AlwaysVersion3 failure

[directhex]

Mono and CoreCLR both fail near-identically on this test on Helix.

CoreCLR https://helix.dot.net/api/2019-06-17/jobs/4f122c80-6e43-48c3-9793-d5c72dc59bbd/workitems/System.Security.Cryptography.X509Certificates.Tests/console

Warning: unable to build chain to self-signed root for signer "(null)"
    System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestUsageTests.AlwaysVersion3 [FAIL]
      Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/X509Pal.cs(192,0): at Internal.Cryptography.Pal.X509Pal.AppleX509Pal.GetCertContentType(ReadOnlySpan`1 rawData)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate2.cs(405,0): at System.Security.Cryptography.X509Certificates.X509Certificate2.GetCertContentType(ReadOnlySpan`1 rawData)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/AppleCertificatePal.cs(80,0): at Internal.Cryptography.Pal.AppleCertificatePal.FromBlob(ReadOnlySpan`1 rawData, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/CertificatePal.cs(37,0): at Internal.Cryptography.Pal.CertificatePal.FromBlob(ReadOnlySpan`1 rawData, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate.cs(63,0): at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(ReadOnlySpan`1 data)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate.cs(52,0): at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(Byte[] data)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate2.cs(51,0): at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(698,0): at System.Security.Cryptography.X509Certificates.CertificateRequest.Create(X500DistinguishedName issuerName, X509SignatureGenerator generator, DateTimeOffset notBefore, DateTimeOffset notAfter, ReadOnlySpan`1 serialNumber)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(577,0): at System.Security.Cryptography.X509Certificates.CertificateRequest.Create(X500DistinguishedName issuerName, X509SignatureGenerator generator, DateTimeOffset notBefore, DateTimeOffset notAfter, Byte[] serialNumber)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(314,0): at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CertificateRequestUsageTests.cs(365,0): at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestUsageTests.AlwaysVersion3()

Mono https://helix.dot.net/api/2019-06-17/jobs/6648e68b-974b-4787-9345-e0223c696d09/workitems/System.Security.Cryptography.X509Certificates.Tests/console

    System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestUsageTests.AlwaysVersion3 [FAIL]
      Interop+AppleCrypto+AppleCommonCryptoCryptographicException : Unknown format in import.
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/X509Pal.cs(196,0): at Internal.Cryptography.Pal.X509Pal.AppleX509Pal.GetCertContentType(ReadOnlySpan`1 rawData)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate2.cs(405,0): at System.Security.Cryptography.X509Certificates.X509Certificate2.GetCertContentType(ReadOnlySpan`1 rawData)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/AppleCertificatePal.cs(80,0): at Internal.Cryptography.Pal.AppleCertificatePal.FromBlob(ReadOnlySpan`1 rawData, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/Internal/Cryptography/Pal.OSX/CertificatePal.cs(37,0): at Internal.Cryptography.Pal.CertificatePal.FromBlob(ReadOnlySpan`1 rawData, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate.cs(63,0): at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(ReadOnlySpan`1 data)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate.cs(52,0): at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(Byte[] data)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate2.cs(51,0): at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(Byte[] rawData)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(698,0): at System.Security.Cryptography.X509Certificates.CertificateRequest.Create(X500DistinguishedName issuerName, X509SignatureGenerator generator, DateTimeOffset notBefore, DateTimeOffset notAfter, ReadOnlySpan`1 serialNumber)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(577,0): at System.Security.Cryptography.X509Certificates.CertificateRequest.Create(X500DistinguishedName issuerName, X509SignatureGenerator generator, DateTimeOffset notBefore, DateTimeOffset notAfter, Byte[] serialNumber)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs(314,0): at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
        /_/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CertificateRequestUsageTests.cs(365,0): at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestUsageTests.AlwaysVersion3()
        /_/src/mono/netcore/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs(384,0): at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Warning: unable to build chain to self-signed root for signer "(null)"

Unknown format in import - has the export format from macOS changed?

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
See info in area-owners.md if you want to be subscribed.

[danmoseley]

Do we need any change here in the Sept servicing? That is about to be code complete (well it was last Friday). I'm guessing that the Sept patch may be the latest available whan macOS 11 releases.

[vcsjones]

I can look at this soon since I have Big Sur available and was looking in to #39603.

[danmoseley]

Many thanks @vcsjones ! It's a busy day for us today with the 6.0 branching.

[vcsjones]

I can repro it; which is good. This is new, this wasn't failing in beta 2. Sigh.

[vcsjones]

@bartonjs okay, so the issue is that Big Sur rejects X509 certificates that have an Extensions SEQUENCE that has 0 elements.

For example:

-----BEGIN CERTIFICATE-----
MIIBXDCB46ADAgECAggfU1S5cWedTzAKBggqhkjOPQQDAzAUMRIwEAYDVQQDEwlU
ZXN0IENlcnQwHhcNMjAwODE3MTcyMDQwWhcNMjAwODE3MTgyMDQwWjAUMRIwEAYD
VQQDEwlUZXN0IENlcnQwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAAQfusjuvQy/NWQL
Oe/ggI3XdN6/8goqMp6RcTuvfX88PoFUbYg3ML7n5IZ4+FewLKDrITEDvWjONDNl
qKTD1FVfo4X1MwIDvddv+tHzr/uVdRwTIAfhskA1PLCkzxaTvfmjAjAAMAoGCCqG
SM49BAMDA2gAMGUCMQDp0320DhrroQ3jTNmjBN2LY00Dja4qEKMt67HvjZ7Rf1pY
qkDnRfa/+FXz9CFy2V0CMGME17+ZX2nRovSjUOrjOiXgz1si3MI8fdaUbCZXM2Vz
wQfSz1Ri6JdqbemG/IxK6w==
-----END CERTIFICATE-----

That certificate will fail on Big Sur:

> security verify-cert -c ~/foo.pem                          
SecCertificateCreateFromData: Unknown format in import.

RFC 5280 says

If present, this field is a SEQUENCE of one or more certificate extensions

Emphasis mine.

I think then, we should change the CertificateRequest to completely omit the extensions [3] from the TBS if there are zero extensions when writing the ASN.

We can't really do a fix-up when loading certificates to allow the old behavior because we'll invalidate the signature on the cert.

Thoughts?

[vcsjones]

Actually, it looks like this was considered and the current behavior is intentional:

runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs

Lines 661 to 663 in 3b824aa

	// An interesting quirk of skipping null values here is that
	// Extensions.Count == 0 => no extensions
	// Extensions.ContainsOnly(null) => empty extensions list

I feel like there is a puzzle piece that I am missing.

[bartonjs]

I think the comment is really just saying "we have a slightly weird behavior", the tests seem to be exercising that, just to prove they know what's what:

runtime/src/libraries/System.Security.Cryptography.X509Certificates/tests/CertificateCreation/CertificateRequestUsageTests.cs

Lines 363 to 382 in 7918347

	request.CertificateExtensions.Add(null);
	using (X509Certificate2 cert = request.CreateSelfSigned(now, now.AddHours(1)))
	{
	Assert.Equal(3, cert.Version);
	Assert.Equal(0, cert.Extensions.Count);
	}
	request.CertificateExtensions.Clear();
	request.CertificateExtensions.Add(
	new X509SubjectKeyIdentifierExtension(
	request.PublicKey,
	X509SubjectKeyIdentifierHashAlgorithm.Sha1,
	false));
	using (X509Certificate2 cert = request.CreateSelfSigned(now, now.AddHours(1)))
	{
	Assert.Equal(3, cert.Version);
	Assert.Equal(1, cert.Extensions.Count);
	}

Probably the ToArray() needs to be guarded by extensionAsns being non-empty:

runtime/src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs

Lines 680 to 681 in 3b824aa

	tbsCertificate.Extensions = extensionAsns.ToArray();

and that comment deleted.

[bartonjs]

(It was probably a semi-bug that was kept as an "interesting test condition", and now is a "doesn't work on macOS")

The 2012 version of X.509 (the ITU spec) didn't contain the one-or-more clause for certificate extensions (it did for CRL extensions). I don't see that being fixed in any of the addenda/errata, but the 2016 version does state it (in section 7.2):

The extensions component, when present, shall hold one or more extensions as defined in clause 7.3.

[vcsjones]

It was probably a semi-bug that was kept as an "interesting test condition", and now is a "doesn't work on macOS"

Makes sense. This seemed like it could have been one of those "we're trying to emulate the behavior of Certificate Services from Windows Server 2000" or something. Your compat concerns are starting to rub off on me.