cargo from 1.56.0 with openssl-3 fails to get ansi_term, goes to amazon

[zarniwhoop73]

Problem

I'm on an experimental fresh build of linuxfromscratch using openssl-3 with llvm-13.0 and rust-1.56.0 (specifically, this system has never had openssl-1.1.1 installed). When I try to build cbindgen, it reports

Updating crates.io index
Downloading crates ...
error: failed to download from
https://crates.io/api/v1/crates/ansi_term/0.11.0/download

Caused by:
  [60] SSL peer certificate or SSH remote key was not OK (SSL
certificate problem: unable to get local issuer certificate)

Other packages (e.g. librsvg) seem to download crates without problems.

I had expected a successful download of all the required crates. I went back to a slightly older system on the same machine using openssl-1.1.1, wiped out ~/.cargo and succeeded in downloading the crates and compiling cbindgen.

Came back to the experimental system, again wiped out ~/.cargo, then ran strace to try to work out if I was missing a certificate (unlikely) and was surprised by what I eventually saw.

First it looks for certs and finds them in /etc/pki/tls/ca-bundle.crt, and it also correctly uses /etc/ssl. Then it looks at the builder's .cargo/registry (empty) and starts downloading registry items including ansi_term. OK so far.

Then it reads the builder's (empty) .cargo/.package-cache and downloads crates. But after getting several of them (ansi_term has not been mentioned among these downloads) it eventually connects to another host:
connect(7, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("99.86.119.114")}, 16) = -1
EINPROGRESS (Operation now in progress)

and after a little while it reports
poll([{fd=7, events=POLLPRI|POLLOUT|POLLWRNORM}], 1, 0) = 0 (Timeout)

After that it cleans up and reports it could not validate the certificate or remote key.

That address is an Amazon cloud host. I do not understand why that is being accessed, and I suspect it happened to be offline or else not offering the sought service.

Something similar (but without strace showing an excursion to Amazon) was reported in kisslinux/repo#339 which suggested that the cert details were somehow being lost when using openssl-3, and that using envvars to point to them fixed it. Did not fix it for me.

This appears to not be true for everyone using openssl-3 (one of my colleagues was able to build cbindgen), but for me it is a showstopper.

Steps

untar cbindgen-0.20.0
cd into cbindgen-0.20.0
run 'cargo build'

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.56.0
release: 1.56.0

[ehuss]

The files from the crates.io registry are hosted on static.crates.io which is a cloudfront CDN. I believe that requires the "Amazon Root CA" certificates. Do you have those installed?

[zarniwhoop73]

Thanks for the quick response. I have

Certificate "Amazon Root CA 1"

Issuer: CN=Amazon Root CA 1,O=Amazon,C=US

Serial Number:06:6c:9f:cf:99:bf:8c:0a:39:e2:f0:78:8a:43:e6:96:36:5b:ca

Subject: CN=Amazon Root CA 1,O=Amazon,C=US

Not Valid Before: Tue May 26 00:00:00 2015

Not Valid After : Sun Jan 17 00:00:00 2038

#Certificate "Amazon Root CA 2"

Issuer: CN=Amazon Root CA 2,O=Amazon,C=US

Serial Number:06:6c:9f:d2:96:35:86:9f:0a:0f:e5:86:78:f8:5b:26:bb:8a:37

Subject: CN=Amazon Root CA 2,O=Amazon,C=US

Not Valid Before: Tue May 26 00:00:00 2015

Not Valid After : Sat May 26 00:00:00 2040

#Certificate "Amazon Root CA 3"

Issuer: CN=Amazon Root CA 3,O=Amazon,C=US

Serial Number:06:6c:9f:d5:74:97:36:66:3f:3b:0b:9a:d9:e8:9e:76:03:f2:4a

Subject: CN=Amazon Root CA 3,O=Amazon,C=US

Not Valid Before: Tue May 26 00:00:00 2015

Not Valid After : Sat May 26 00:00:00 2040

Certificate "Amazon Root CA 4"

Issuer: CN=Amazon Root CA 4,O=Amazon,C=US

Serial Number:06:6c:9f:d7:c1:bb:10:4c:29:43:e5:71:7b:7b:2c:c8:1a:c1:0e

Subject: CN=Amazon Root CA 4,O=Amazon,C=US

Not Valid Before: Tue May 26 00:00:00 2015

Not Valid After : Sat May 26 00:00:00 2040

and Trust for each of them. That is from what mozilla use, I hope that is all the amazon root certs ?

[alexcrichton]

I'm able to reproduce locally and I think I see the bug. With custom OpenSSL setups Cargo tries to probe the system and find ssl certificates and I suspect that the version of OpenSSL you've got installed in your system probably isn't configured or something is going wrong such that it's not finding the system certificates.

Typically this is ok since Cargo informs of the system certificates via the openssl-probe crate. The problem (at least for me locally) was that the curl crate was initializing openssl manually before openssl-probe set env vars which meant that openssl cached state which wasn't re-read later when the env vars were set.

The reason that this works today for me with 1.1.1 build statically is that #[cfg(need_openssl_init)] prevents curl from initializing openssl, which delays openssl's initialization until after openssl-probe has run. That cfg isn't set because this block doesn't print out a version directive (presumably b/c it was so new when it was added).

So basically I think the issue is probably two things:

1. Your system build of OpenSSL may not be configured such that it can natively find the system certificates without any extra help. I'm mainly guessing this because things otherwise aren't working.
2. There's two bugs in crates:

• In openssl-sys it's not printing a version directive for 3.0.0
• In curl it's initializing openssl before using openssl probe functions to set env vars (unlike the git2 crate which initializes env vars before initializing libgit2 which initializes openssl)

I think fixing any of the above points should be sufficient for fixing your issue.

[zarniwhoop73]

I don't understand why you think my openssl build is misconfigured. I've been using https:// with falkon and seamonkey when the experimental system was booted. At the moment the new system is in chroot building other things, and there I've now used https:// with links.

OTOH, with both links in chroot and firefox on the older host system, https:// 9.86.119.114 hangs (the older host system is openssl-1.1.1 where cargo-1.52.0 builds cbindgen without problems)

[zarniwhoop73]

I found an example of verifying certificates on a redhat page, using the command 'echo | openssl s_client -connect redhat.com:443 -brief'

In both my host system and in chroot on the experimental system the output includes 'Verification: OK' so I'm fairly sure my openssl setup is ok.

[zarniwhoop73]

Sorry, I'm a bit slow, trying to do other things. I've just tried that command on 99.86.119.114:443 and both the 1.1.1l and 3.0.0 systems report a version problem.

With openssl-1.1.1l:
echo | openssl s_client -connect 99.86.119.114:443 -brief
139867656247104:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:331:

and with 3.0.0
echo | openssl s_client -connect 99.86.119.114:443 -brief
4037C6A4637F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:354: