Upgrade to hyper-rustls 0.27/rustls 0.23

[djc]

Sticks with ring as the default crypto provider for rustls for now (see #2136).

This is also a prerequisite for re-enabling a Quinn-based H3 backend, since Quinn is skipping rustls 0.22.

[seanmonstar]

Thanks so much! I do have some questions, to help plan how we could allow using the different crypto providers.

• If no provider is enabled, what happens? I tried removing "ring" from the dependencies in this PR locally, and it still builds...
  • Oh I see, I just ran the simple example, it panics at runtime.
• I suppose we could introduce rustls-tls-ring and rustls-tls-awslc features eventually. We don't need to do so yet, but is there anything to take into consideration?

[djc]

Thanks so much! I do have some questions, to help plan how we could allow using the different crypto providers.

• If no provider is enabled, what happens? I tried removing "ring" from the dependencies in this PR locally, and it still builds...

  • Oh I see, I just ran the simple example, it panics at runtime.

Yes. To be safe at compile time, we could use the builder_with_provider() API and pass rustls::crypto::ring::default_provider() explicitly instead.

• I suppose we could introduce rustls-tls-ring and rustls-tls-awslc features eventually. We don't need to do so yet, but is there anything to take into consideration?

Those features would be non-additive, because rustls will panic if both are enabled. Also note that people might want to use a third-party crypto provider -- this is of course already possible via the preconfigured_tls() path.

[seanmonstar]

Those features would be non-additive, because rustls will panic if both are enabled.

Oh, so reqwest would be picking the crypto provider for the application as a whole? If any other library depends on rustls and doesn't disable the default features, will the application start to see panics when they upgrade reqwest?

[djc]

The ClientConfig::builder() indirectly calls into here:

https://github.com/rustls/rustls/blob/main/rustls/src/crypto/mod.rs#L254

This will:

1. If a process-wide crypto provider has already been installed, use it
2. If no provider has been installed and only one has been enabled via Cargo features, use it and install it as the default
3. Otherwise, panic

As such, I think the idea is that reqwest should rely on this and if people run into a run-time panic they should call CryptoProvider::install_default() first before setting up reqwest's Client/ClientBuilder.

See more discussion in rustls/rustls#1877.

[NobodyXu]

Won't it be more reasonable, to prefer one crypto to another instead of panicking at runtime?

Or maybe a compile_error! is also better than runtime panic.

[djc]

Feel free to contribute to the discussion in the upstream issue.

[stormshield-gt]

I would like to suggest a change to be able to setup a custom provider without having to compile both ring and the custom provider.

• do not activate the ring feature on hyper-rustls, rustls and tokio-rustls
• activate the ring feature only if rustls-tls feature is enabled. rustls-tls = ["rustls-tls-webpki-roots", "rustls/ring"]
• optionally, add another feature that uses rustls but don't activate the ring feature. rustls-tls-providerless = ["rustls-tls-webpki-roots"]. This is optional as users could use the rustls-tls-webpki-roots directly

[djc]

FWIW, this PR was just intended to do the rustls upgrade while making as few other changes as possible. Obviously there could be a number of new Cargo features that help control the crypto provider used by rustls. But I think reqwest will want to pick a default crypto provider to make adopting rustls as easy as possible.

[stormshield-gt]

My suggested changes still pick up ring as a default provider without adding any new feature, and leaving the possibility to control the crypto provider used by rustls. That's why I think it's interesting to consider this now. This can also be, of course, in a follow-up PR as you suggest.

[seanmonstar]

I do want this to merge. As to the status of this upgrade: I'm hesitant to merge and publish since anyone upgrading that was already using rustls v0.23 may suddenly start seeing a panic, because reqwest enables the alternative backend. (And I don't really want to "fix" that by switching to aws-lc.)

My feeling is that I'd want to wait for a rustls version that doesn't have that issue. Discussion is in rustls/rustls#1877.

[djc]

@seanmonstar I think we can work around that issue by using ClientConfig::builder_with_provider(), explicitly passing whichever provider we have selected into the config builder explicitly. This should never panic, independent of which Cargo features other libraries have selected. Does that sound like a suitable mechanism for now?

I feel like a solution to rustls/rustls#1877 is unlikely to happen soon.

[djc]

I've revised this to explicitly select a ring (using the existing Cargo features) or aws-lc-rs (using a parallel set of Cargo features) provider and use the selected crypto provider explicitly when constructing a rustls ClientConfig. This setup should also enable preventing a dependency on either ring or aws-lc-rs by using a new rustls-base feature (renamed from __rustls) -- the latest release depends on ring (via rustls) unconditionally so even though one could wire up a preconfigured rustls ClientConfig using aws-lc-rs or another crypto provider, there would still be a ring dependency.

This is motivated in part by rustup concerns for making rustls the default TLS implementation, since some scenarios need P521 cipher suites which only aws-lc-rs supports.

[djc]

So the MSRV build fails with this error:

error: Package reqwest v0.12.4 (/Users/djc/src/reqwest) does not have feature rustls. It has an optional dependency > with that name, but that dependency uses the "dep:" syntax in the features table, so it does not have an implicit feature > with that name.

Not entirely sure what's up with that? From looking at the state of Cargo.toml after these changes I don't think there are any references to a rustls feature left?

(Seems to work okay with current stable.)

[asonix]

fwiw just running cargo c on this branch with rust 1.63 results in the same error (ubuntu 24.04 container)

[asonix]

Suggested change

	__rustls_crypto_ring = ["rustls-base", "rustls/ring"]
	__rustls_crypto_aws_lc = ["rustls-base", "rustls/aws_lc_rs"]
	__rustls_crypto_ring = ["rustls-base", "rustls?/ring"]
	__rustls_crypto_aws_lc = ["rustls-base", "rustls?/aws_lc_rs"]

a few well-placed questionmarks seem to allow this to work on 1.63

[asonix]

hyper-rustls 0.27.1 also bumps msrv from 1.63 to 1.64

[asonix]

Suggested change

	feature = "__rustls_crypto_aws_lc-rs"
	feature = "__rustls_crypto_aws_lc"

Feature name is wrong

[asonix]

Suggested change

	feature = "__rustls_crypto_aws_lc-rs",
	feature = "__rustls_crypto_aws_lc",

feature name is wrong here too

[asonix]

Suggested change

	feature = "__rustls_crypto_aws_lc-rs"
	feature = "__rustls_crypto_aws_lc"

and here

[asonix]

Suggested change

	feature = "__rustls_crypto_aws_lc-rs"
	feature = "__rustls_crypto_aws_lc"

here

[asonix]

Suggested change

	feature = "__rustls_crypto_aws_lc-rs"
	feature = "__rustls_crypto_aws_lc"

at this point it's probably easier to add -rs to Cargo.toml

[asonix]

actually I'd suggest changing this just to rustls-base since it doesn't compile with only that feature otherwise

[asonix]

Either that or gate the Debug impl behind this same cfg(any())

[asonix]

From my look into this PR, I think TlsBackend can't have a Default impl if only rustls-base is enabled, since the TlsBackend::Rustls variant indicates that either aws-lc-rs or ring has been selected, and requires the call to default_provider() for one of the backends.

In that scenario it is required that a user pass in a a rustls-compatible clientconfig so that the BuildRustls variant is set on the client.

Would it make sense to set UnknownPreconfigured as the default in that case? That could lead to confusing error messages, though, so maybe a new variant UnsetRustls or something could be created in order to tell users that rustls was enabled but no backend was configured

[asonix]

I think these compile issues could have been caught by CI if the new features were added to the github actions workflow, also

[seanmonstar]

Alright, I've returned to this. I do want to get an upgrade out. I think the recent review comments do catch some typos, which would be good to fix.

As I understand the code now, this will explicitly use the crypto provider enabled with features, and will not ask for the "default" from rustls. The downsides that still exist for releasing this upgrade are:

• Users who have an app already depending on rustls 0.23, default features, and are using ClientConfig::builder() or ServerConfig::builder() will start to see a panic when pull this in. (It seems like those constructors are mostly a bad idea to use, in general.) It might appear like reqwest's "fault", but there's not much we can do there.
• It looks like the features within reqwest are not additive. If someone enables the rustls-tls-aws-lc-webpki-roots feature, and another dependency has enabled rustls-tls (which still implies ring), all reqwest::Clients will use the ring provider. Would we need to provide builder methods to pick at runtime?

[Alvenix]

For reference for another approach you can check the approach which we used in Rocket. Essentially the crypto provider by Rocket is chosen like this.

fn default_crypto_provider() -> CryptoProvider {
    // Use the provider chosen by the user at runtime
    CryptoProvider::get_default()
        .map(|arc| (**arc).clone())
         // if the user didn't select any at runtime, use ring provider
        .unwrap_or_else(ring::default_provider)
}

Rocket have single feature for TLS which enable ring as it currently have simpler build requirements. If the user want to change the CryptoProvider he can add the following at the start of the main for example:

fn main() {
    rustls::crypto::aws_lc_rs::default_provider().install_default().unwrap();
}

It had less features for TLS which is simpler but the cost of this is that if the user want to aws_ls_rs he would have to do at runtime.

The approach in this PR is more flexible due to features being used.

[asonix]

I see that a simple PR with rustls 0.23 has been merged. I would still be interested in more configurability here. When upgrading to rustls 0.23 i've migrated my other dependencies from ring to aws-lc-rs, and I will be providing an aws-lc-rs ClientConfig to reqwest in my application, so I would like to avoid compiling ring if possible

[seanmonstar]

As mentioned, a subset of this has been merged in #2299.

You can open an issue about aws-lc features, if you'd like to discuss. One thing that's important to me in reqwest is that features be additive, and we maintain backwards compatibility.

[djc]

Note that with the merged PR you can still use aws-lc-rs by installing a default provider before initializing the Client.