Wrong Let's Encrypt CA chain presented by your server

[lastuptodate]

Frequently asked questions

• I have read Frequently Asked Questions

Describe the bug
packages.sury.org web server is presenting wrong CA chain
see: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

To Reproduce
Steps to reproduce the behavior:

openssl s_client -connect packages.sury.org:443
Certificate chain
 0 s:CN = packages.sury.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

# update-ca-certificates 
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
# apt update
Hit:1 http://repo.clacos.ninja buster InRelease
Ign:2 https://packages.sury.org/php buster InRelease                                                                                                                                                              
Err:3 https://packages.sury.org/php buster Release                                                                                                                                                                
  Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate.  Could not handshake: Error in the certificate verification.

Expected behavior
The certicate must be recreated with the right CA chain, see: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
This is generaly the case when you are using an outdated certbot provided by Debian rather than certbot by https://certbot.eff.org/. In some cases, you have to delete+create cert rather than renew. Also, --prefered-chain may also help you.

Distribution (please complete the following information):

• OS: Debian
• Architecture: amd64
• Repository: packages.sury.org

[oerdnj]

I don't really see the behaviour described here:

ondrej@kage:~$ gnutls-cli packages.sury.org:443
Processed 189 CA certificate(s).
Resolving 'packages.sury.org:443'...
Connecting to '185.152.64.17:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `CN=packages.sury.org', issuer `CN=R3,O=Let's Encrypt,C=US', serial 0x0391013a17469f7f084671ef20c765694d6e, RSA key 2048 bits, signed using RSA-SHA256, activated `2022-05-22 10:10:15 UTC', expires `2022-08-20 10:10:14 UTC', pin-sha256="jgwPBRVN5iMySczBW4m1VqMzCmw/ETYRRT2jL+pBZXA="
	Public Key ID:
		sha1:ed56dceb8acbaa62da327a6400abcc40b66540e7
		sha256:8e0c0f05154de6233249ccc15b89b556a3330a6c3f113611453da32fea416570
	Public Key PIN:
		pin-sha256:jgwPBRVN5iMySczBW4m1VqMzCmw/ETYRRT2jL+pBZXA=

- Certificate[1] info:
 - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
- Certificate[2] info:
 - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US', issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30 18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
- Status: The certificate is trusted.
- Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Session ID: 6F:CD:59:92:B7:AA:8C:1B:3D:07:E4:F2:A0:61:65:9E:F8:DB:5D:3F:F7:64:B9:75:9C:AC:A5:E5:B4:8E:C7:2A
- Options:
- Handshake was completed

- Simple Client Mode:

- Peer has closed the GnuTLS connection

$ openssl s_client -connect packages.sury.org:443
CONNECTED(00000005)
depth=3 C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
verify return:1
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = *.b-cdn.net
verify return:1
---
Certificate chain
 0 s:/CN=*.b-cdn.net
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA
   i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
---
Server certificate

So, what is the DNS records and IP address(es) you are connecting to?

[oerdnj]

Nor with SNI:

$ openssl s_client -connect packages.sury.org:443 -servername packages.sury.org
CONNECTED(00000005)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = packages.sury.org
verify return:1
---
Certificate chain
 0 s:/CN=packages.sury.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFKTCCBBGgAwIBAgISA5EBOhdGn38IRnHvIMdlaU1uMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjA1MjIxMDEwMTVaFw0yMjA4MjAxMDEwMTRaMBwxGjAYBgNVBAMT
EXBhY2thZ2VzLnN1cnkub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAlLjyBAnQJuBcJpJqgOMmAkNSK8yigld1prP4wljNoGnYXuRWQ3uPcpcSCeQC
yT3b5xg/ZkQG0KtSlIZCV6R/8zqGFU/tj0wqZYh6sOxKzJGjIPPjbHRu9aZ/LgT8
f1GECT42+SDarTkDlX16Oggj3q69hEFgsAs9QF4eb8mSINi6X8oKW8vcYVElWZyd
c8WbkvSUwtwEqKLpXPkNvECloReANeyVCNZ0eh48PQOvwQ8vkwglojyerNpK85Rm
mHbDj6/4TMp9NwwfW50BYqL/787c2iHShyxcqAl8pPVXnTmuznNpMEm5OVa8Ch4M
v+FSfonzYy4+d8vZvtcDcF2VBwIDAQABo4ICTTCCAkkwDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBQy43kyKEjVoswPhP2fTqIYhOCsbzAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghFwYWNrYWdlcy5zdXJ5Lm9yZzBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYEgfMA8QB3AEHIyrHf
IkZKEMahOglCh15OMYsbA+vrS8do8JBilgb2AAABgOt17W4AAAQDAEgwRgIhANfo
CD2reLy+rGOLbEzJRKkTYm/PMO/bJkSjC5gw4PzyAiEAgu8xyAS79AW0AZNjAP+b
fepkRRKpLzWTL2kHVNXx7N4AdgApeb7wnjk5IfBWc59jpXflvld9nGAK+PlNXSZc
JV3HhAAAAYDrde1qAAAEAwBHMEUCIETybellG1YyDh6YXLohH4gzoAcLVuMCtR/p
aYRuNZ6IAiEA0iQ6N7NyLQAm/MoedLiqXTUIPEmcgyp5hGA/3QW1HZswDQYJKoZI
hvcNAQELBQADggEBAKW/uGoAHFUGVsqOb1mU6Jnc9psr5d4HlJVvXPSut/F3gyM3
+5JW0UJyG3KyPatlHVnEvdbBu3YF8aUxUsoFB+IKtBEL+GIKxyvxU1HojDYM2Lt6
TZt6RRlPVoWxIgSOX8jcQdvreJawTI7i8AOXJ59ZgK5DsLmzA0dXJfQYdFQAeEQX
0dtE7Qf4y1I5ZKHQhrydoTpObuk3qkT2GSN1CrQC9jHmHGjEX5ReN1V7Mo9c5hp/
opieB4jTpVhnNNQBKtNGIke8ySdGRoOIrOiCITrOMkE/sxIwzx0PE2Afjp2OWyh5
shQBHJJslbSODPNyOQsQDxnLdmnsumnU0XoTzJI=
-----END CERTIFICATE-----
subject=/CN=packages.sury.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 4666 bytes and written 307 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 39FD7FAC3DB1FBD7265266E55B982E5F057AAB1E02D66A952276AE77D2D39507
    Session-ID-ctx:
    Master-Key: D01418236C4F792C43DE402B14DC8EE0794816990BF0FECACEEDC3FBD83D92724F40AEF848A8E919517C000B9A4915E1
    TLS session ticket lifetime hint: 43200 (seconds)
    TLS session ticket:
    0000 - de af 11 c7 7f df 57 0b-7e d8 66 63 ab ac 3e e7   ......W.~.fc..>.
    0010 - 8c e6 17 5a 5c f2 59 2d-1c f9 93 11 0e 12 2e 7d   ...Z\.Y-.......}
    0020 - 26 f8 6a ef 02 4a 33 a6-d3 d9 59 71 a1 f7 4b 59   &.j..J3...Yq..KY
    0030 - 42 a4 91 40 2b 98 08 23-9c cb 28 bb 8b 86 6d b8   B..@+..#..(...m.
    0040 - d6 fd 7e 94 61 e6 a7 9f-1d e6 14 4b 0d 01 7f 32   ..~.a......K...2
    0050 - 71 6f 30 62 b0 bc 55 e8-1a 77 5f 0b 77 d7 58 86   qo0b..U..w_.w.X.
    0060 - 35 b4 d4 cc 53 4b 4f 9f-91 4b 7c fb 6a 03 15 ff   5...SKO..K|.j...
    0070 - ef 98 1d 91 43 e4 80 98-31 f4 8f 7c d2 e5 83 b0   ....C...1..|....
    0080 - a2 53 25 17 ac 85 51 1b-28 31 e6 a9 a0 45 f4 43   .S%...Q.(1...E.C
    0090 - 34 37 8b 66 4e f4 1a 00-8d 53 3b 89 c1 d6 64 68   47.fN....S;...dh
    00a0 - 53 b1 b6 89 52 74 75 dd-e5 6b f3 89 bf 7d 60 95   S...Rtu..k...}`.
    00b0 - eb d6 ed 52 6e 11 db 67-28 59 b9 8a 6f 79 60 d8   ...Rn..g(Y..oy`.

    Start Time: 1655364564
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

[oerdnj]

Neither there's anything wrong at the SSLLabs: https://www.ssllabs.com/ssltest/analyze.html?d=packages.sury.org

There's trusted Certification Paths listed for every client.

[oerdnj]

Nevertheless, I opened the ticket with @BunnyWay to remove the cruft being sent as part of the certificate chain.

However, if the validation fails on your side, it means you are missing:

ISRG Root X1 Self-signed
Fingerprint SHA256: 96bcec06264976f37460779acf28c5a7cfe8a3c0aae11a8ffcee05c0bddf08c6
Pin SHA256: C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=
RSA 4096 bits (e 65537) / SHA256withRSA

in the trust store.

[oerdnj]

But perhaps it's intentional - if you read the link that you sent yourself:

DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates. There’s one important exception: older Android devices that don’t trust ISRG Root X1 will continue to work with Let’s Encrypt, thanks to a special cross-sign from DST Root CA X3 that extends past that root’s expiration. This exception only works for Android.

I don't really care about the Android devices, but other CDN users might.

[lastuptodate]

Let's encyrpt has changed CA chain

• If your server is presenting Leaf (packages.sury.org) + R3, the ssl client (apt, firefox etc) will verify the R3 cert with ISRG Root X1 in it's local CA cert store. This is the good and recommanded updated path

• if your server is presenting the OLD chain: Leaf + R3 + ISRG X1, the ssl client will verify ISRG X1 with DST Root CA X3 in it's local store, wich is expired. Some client are clever and can re-assemble the short path (without Root CA X3), but some client like old apt are using the long path with the expired cert.
  If you have a look on sslabs, you have 2 Certifcations Paths. Path1 is Trusted but Path2 is untrustesd. Depending with we are using old or new ssl client, we will have problems using path 2

In this example, we are using LE with a unique and good CA chain path. This is the recommanded solution

[sleemanj]

@oerdnj is correct, it's intentional, this is the cross-signed chain that servers should serve up so that (older) Android doesn't get lost (with that said, how many Androiders are going to be pulling the repo).

Submitter must be missing the ISRG root, which they can get here...
https://letsencrypt.org/certs/isrgrootx1.pem

[todeveni]

@lastuptodate What's your ca-certificates package version?

buster without ca-certificates installed:
- Status: The certificate is NOT trusted. The certificate issuer is unknown.

buster with latest ca-certificates 20200601~deb10u2 installed:
- Status: The certificate is trusted.

[oerdnj]

@lastuptodate Well, you are wrong. The very material you provided disputes what you are saying. There's nothing wrong with the certificate chain sent from the server. Thanks @sleemanj for confirming my hypothesis.

[lastuptodate]

Have a look at CA Chain, they are both using Let's Encrypt:
https://www.hardenize.com/report/packages.sury.org/1655367513#www_certs
https://www.hardenize.com/report/vdp.gong.io/1655367866#www_certs

[oerdnj]

Have a look at CA Chain, they are both using Let's Encrypt:
https://www.hardenize.com/report/packages.sury.org/1655367513#www_certs
https://www.hardenize.com/report/vdp.gong.io/1655367866#www_certs

#1792 (comment)

[lastuptodate]

Thank for your help and your time, I think sadly that i will not convince you so I will resolved this problem using this sad and worst solution:

Acquire::https::packages.sury.org::Verify-Peer "false";
Acquire::https::packages.sury.org::Verify-Host "false";

[oerdnj]

The hardenize.com report is plain wrong, it shows just the second chain. I've sent you the ssllabs.com report that you chose to ignore that shows both certificate paths.

It's your choice to ignore the facts and advice what's wrong with your system, so don't put it on me.

[sleemanj]

@lastuptodate

If you remove the expired X3 root from your system, and have the valid ISRG root on your system, then I expect it might verify.

You just need one of them to verify, if as you say your tools are picking the X3, then you want to convince them to pick the ISRG, since the X3 is no use to you on your system (but is to old Androiders on their system).

I think, from vague recollection, that's what I did back last year, on my ancient 14.04 (not for pulling this repo obviously, but to shut up some other stuff).

---

Indeed, looking in /etc/ca-certificates.conf on that 14.04 system I have the X3 cert disabled.

[todeveni]

@lastuptodate

I'd still like to add, that your local certificate store is old and/or badly maintained.

There's zero problems with clean Buster docker image using the instructions in README.txt, which installs latest ca-certificates package.

[lastuptodate]

I can't also reproduce on a fresh Buster but I can on old Buster install.
So I try to see in "apt list --upgradable" packages linked to apt/curl or ssl.

And the winner is:

Unpacking libgnutls-dane0:amd64 (3.6.7-4+deb10u7) over (3.6.7-4+deb10u3) ...
Unpacking libgnutls30:amd64 (3.6.7-4+deb10u7) over (3.6.7-4+deb10u3) ...

Upgrading this 2 packages solved my problem.
Thank you all and really really really sorry and all my apologies for the noises about that

[akaPipo]

I have fixed it this way #1729 (comment)