Cluster creation failed with certificate validation error when using mirror registry with self-signed certificate

[rimaulana]

What happened:

Failed in validation step during cluster creation due to invalid registry certificate when using self-signed certificate and provided the registryMirrorConfiguration.caCertContent field

Warning: The recommended number of control plane nodes is 3 or 5
Performing setup and validations
✅ Connected to server
✅ Authenticated to vSphere
✅ Datacenter validated
✅ Network validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Control plane and Workload templates validated
✅ Vsphere Provider setup is valid
❌ Validation failed     {"validation": "create preflight validations pass", "error": "validation failed with 1 errors: invalid registry certificate: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0", "remediation": ""}
Error: validations failed

What you expected to happen:

CA certificate is valid and cluster creation should proceed without any error.

How to reproduce it (as minimally and precisely as possible):
Use registry mirror configuration that has a self-signed certificate and provided the self-signed CA certificate as well in caCertContent field.

Anything else we need to know?:

The CA and server certificate for the registry mirror I was using on harbor was generated using AWS Private CA. When validating server certificate using openssl_client the certificate is valid.

$ openssl s_client -showcerts -connect core.harbor.g4k.io:443 -CAfile ca.crt 
CONNECTED(00000003)
depth=1 C = US, O = AWS, OU = Support, ST = Virginia, CN = core.harbor.g4k.io, L = Herndon
verify return:1
depth=0 CN = core.harbor.g4k.io
verify return:1
---
Certificate chain
 0 s:CN = core.harbor.g4k.io
   i:C = US, O = AWS, OU = Support, ST = Virginia, CN = core.harbor.g4k.io, L = Herndon
-----BEGIN CERTIFICATE-----
MIIDtDCCApygAwIBAgIRAMi3uW9C4cli2HqdkIMQ5h4wDQYJKoZIhvcNAQELBQAw
bzELMAkGA1UEBhMCVVMxDDAKBgNVBAoMA0FXUzEQMA4GA1UECwwHU3VwcG9ydDER
MA8GA1UECAwIVmlyZ2luaWExGzAZBgNVBAMMEmNvcmUuaGFyYm9yLmc0ay5pbzEQ
MA4GA1UEBwwHSGVybmRvbjAeFw0yMjA0MTUwNzE2MTJaFw0yMjA3MTQwODE2MTFa
MB0xGzAZBgNVBAMTEmNvcmUuaGFyYm9yLmc0ay5pbzCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBALmKpff+Oq2j7eB/P//4fQYGMlJA2FRVdm5NPw956bj2
unU7fMSpTiUpFaavRFpRGOSe1vIvSY8eSvz7yfJinbbilPQpwrEtahTmX/ODIan1
ZC6gkfhXEI5VyW1D2WrB/MVibfYVWDFVMD0a/WdgcAWG+crIpFrmHTd85PH0njdD
0gTg6hFMnFjA1L8W5EmY7KhUU9kk9T4sAaH39zflJ8zJYw5HOd2igIKnic+JRU3o
4eS6bGRplO8Xh3Oqkt9pYkuw2NAKyVAgv8DZT+xItULGBSWGTIDGAWykT2RukzRD
nsS4nwmR6NKHnwggjhxOGHaz5g9Aw2KQN+Qs2se4EbkCAwEAAaOBnDCBmTAdBgNV
HREEFjAUghJjb3JlLmhhcmJvci5nNGsuaW8wCQYDVR0TBAIwADAfBgNVHSMEGDAW
gBRH/XsAGH9MQThp1fs2gNafQkZpLjAdBgNVHQ4EFgQUblCMnuhW3f4yq03Q+R5i
NM21PGAwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEF
BQcDAjANBgkqhkiG9w0BAQsFAAOCAQEAFw5fNgGkeFIyu0MPpKGJCpNDcrlafk5/
6cKobCvktD+DGx5WSV9eBq0YXAVjAwnzwyINzrGkGjk9MP7wEjfidWBq3ifikkd3
eCvPNmUcPBs8N24N1Z1eg3xhGFu3ChSwC6iMXMqhpAl/+AsWtj2XWT4NBPkaTfUq
CMCQRcJ7nbIpAw+Jnl7KazRPhoNsunwdObWsGShUYxUlifsLo2O33hBTPcPh++h4
iHFpK8drjEUobQNYiI4xy7iLpYd3eeF1Mi4IaBb+Fx+DQunJ92muGixNBaE6NPLM
urpjBTLkWf3rGLDUQcC9dagAevL3FuHB9IO9I3SeK7q3XYPAQHcu0Q==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = core.harbor.g4k.io

issuer=C = US, O = AWS, OU = Support, ST = Virginia, CN = core.harbor.g4k.io, L = Herndon

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1512 bytes and written 390 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 37239BC3E9B96819FDF3D11C9A3B5928BDCF6C19A4601A112B97DF85CBD5B3C2
    Session-ID-ctx:
    Resumption PSK: 718C939703429E384CA5B35F03BB7FA5764175AD5D47204CDE4CFC8CA2FA9823E6148822D1995FFD9C5304936180AC4E
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 49 69 21 cf fc 92 de 7f-a1 2e cd 9e da 98 b3 04   Ii!.............
    0010 - 04 f0 b8 ae fd 5f 8d 81-15 e0 e7 bd 4f 68 4e 9f   ....._......OhN.

    Start Time: 1650052564
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: F8A514F4C064B862294097DACE434F9F45223D821EF448040FA5B1223FB734C7
    Session-ID-ctx:
    Resumption PSK: 0F3153BEC99E57D32CF7C95472D1E694D01F97C064B59D0D05C813E0B5EA1B26A3CD6ABA0E9ED689FCD93D35FC294ED5
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - c8 d1 c3 45 b1 73 d3 f8-6f 06 0d ed 16 ae 64 9c   ...E.s..o.....d.
    0010 - 85 00 e7 9f 3d 7e 7d e8-5b 24 1b be a6 94 f1 6c   ....=~}.[$.....l

    Start Time: 1650052564
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

It looks to me that the error message was coming from this validation function. I do not believe CA certificate should have sans name and a better way to validate the certificate would be like what golang x509 Certificate.Verify example in which it takes into consideration both CA certificate and server certificate.

I think the test case that was written here is incorrect as the test certificate is the generated server certificate (server.crt) not the CA certificate that signed it. The function should evaluate registryMirrorConfiguration.caCertContent not a certificate that is presented by the server.

I also found out that using a registry with self signed certificate helm template doesn't respect --insecure-skip-tls-verify which causing cillium deployment to fail. A separate issue is opened in helm github page

helm/helm#10868

Environment:

• EKS Anywhere Release: v0.8.2

[g-gaston]

@rimaulana thanks for the awesome bug report! Will look into it and get back to you.

[g-gaston]

Findings:

1. We should change the cert validation to either:

   • Only verify the ca cert format
   • Retrieve the server cert from the registry endpoint and verify it putting the ca cert in the trusted certs (roots)

2. We also need to modify the e2e test data to use the ca.crt. Right now is using the server cert, hence why is not failing.