platform-verifier as feature

[algesten]

On (my) macOS, the rustls-platform-verifier crate introduces 9 dependencies:

> bitflags v2.6.0
> core-foundation v0.9.4
> core-foundation-sys v0.8.7
> num-bigint v0.4.6
> num-integer v0.1.46
> num-traits v0.2.19
> rustls-platform-verifier v0.3.4
> security-framework v2.11.1
> security-framework-sys v2.11.1

ureq tries to keep the number of dependencies low, and without this crate, using default features, have ~48. which means this dep alone is almost a 20% increase in dependencies. Given that ureq2.x shipped with webpki as the defaults roots, we might want to consider feature flagging the rustls-platform-verifier and defaulting to webpki also in 3.x.

(as a side note, someone inspecting Cargo.lock will probably be surprised finding "jni" in their project - even if it's only used on android)

[algesten]

Hey @cpu, I don't imagine there's anything we can do about the dep-count of the rustls-platform-verifier crate, but I wonder if you have some input regarding making webpki the default choice instead of platform-verifier?

I know in some enterprise environments there might be requirement to use the platform roots. webpki crate is also not automatically updated, meaning a long lived service talking to some other service might just stop working without recompilation.

Are there other considerations?

[cpu]

Hi @algesten,

This was a timely question. We're in the process of improving some of the platform verifier docs to hopefully help with this sort of evaluation.

I know in some enterprise environments there might be requirement to use the platform roots. webpki crate is also not automatically updated, meaning a long lived service talking to some other service might just stop working without recompilation.

Are there other considerations?

Yes. With respect to using stale webpki-roots data I think the risk is less about a long-lived service not working due to a peer using certs chaining to a root it doesn't know about yet and more about that service continuing to work when it speaks to a peer using certs chaining to a root that was removed, e.g. due to a security event.

There are also other considerations that make the platform verifier experience a more secure/robust one:

• Webpki & webpki-roots won't help you manage CRLs or revocation state. Microsoft's platform verifier does this in some capacity as an example. Webpki supports loading CRLs but you're on your own for fetching them from somewhere, keeping them up to date, and configuring webpki to use them for revocation checking.
• Webpki & webpki-roots can't express "gradual distrust". E.g. Mozilla are distrusting some Entrust roots, but only for certificates issued after a certain date. We can't express that policy in the webpki-roots ecosystem right now. It's either removed or not. In general we have very limited mechanisms for expressing nuanced policy decisions, it's very binary (with the exception of imposed name constraints, which we do support in webpki-roots).
• Webpki & webpki-roots don't validate requirements related to Certificate Transparency. The platform verifiers can check SCTs in certificates to ensure they were submitted to a CT log. If you're using webpki+webpki-roots you could inadvertently accept a certificate that wasn't logged (e.g. because it was obtained through a compromised CA with the attacker trying to remain undiscovered). Verifying SCTs also allows for another "gradual distrust" mechanisms (e.g. Chrome is distrusting Entrust based on SCT properties).

rusts-native-certs is somewhat of an in-between. It helps with the problem of "stale" root data, but it doesn't help with revocation, gradual distrust, SCTs, etc. It also seems to less robustly support enterprise roots injected into the local trust stores. In most cases we're pushing folks to stop using that crate and to use the platform verifier crate instead.

I suspect there are other advantages but those are the ones top-of-mind for me. Hope that helps!

[algesten]

@cpu Thanks a lot for that input.

I had another thought yesterday, which is where is ureq typically used?. As I see it, although it is a generic http client, ureq is probably mostly used for HTTP API interactions (that's why json is enabled by default). I can only speculate, but I suspect that means the most dominant environment where ureq runs is Linux (inside a Docker container).

For Linux, the platform verifier doesn't do much, right? It uses webpki, no revocations. Is there a big difference between using the cacerts installed by the distro vs the webpki-roots crate?

[cpu]

np!

For Linux, the platform verifier doesn't do much, right? It uses webpki, no revocations. Is there a big difference between using the cacerts installed by the distro vs the webpki-roots crate?

The platform verifier crate on Linux/BSD will end up using rustls-native-certs with webpki, so you get the CA certs installed by the distro (optionally augmented with webpki-roots if you use Verifier::new_with_extra_roots()). That "with extra roots" stuff exists because 1Password was getting some reports that discovering the system roots wasn't always reliable (Some discussion here: rustls/rustls-platform-verifier#12). In general using platform-verifier on UNIX should be more convenient than managing this stuff yourself.

In terms of skipping the platform verifier entirely and choosing between using rustls-native-certs yourself directly, or using webpki-roots, the two important questions are who you trust to keep the source of truth updated & how often you expect to deploy new builds of your app.

If you don't trust us to keep webpki-roots up-to-date (I think we do a good job of this 😆) then you're better off using rustls-native-certs and letting your distro manage that.

If you don't plan to rebuild your application frequently (to pick up new webpki-roots) then you're better off using rustls-native-certs because it will be updated by your distro separate from your application builds/deploys.

Does that make sense?

[cpu]

Tagging @complexspaces to keep me honest. Have I done an OK job summarizing?

[algesten]

@k0nserv you often have strong opinions. What's your take on this?

[algesten]

Another thought: if ureq is mostly used for HTTP APIs, the number of disparate certs the ureq instance would be subjected to is very limited. I think revocations would be an even less concern in a situation where one uses ureq to talk to the same 3-4 HTTP API as part of a service.

If the assumption of ureq use holds, it means the decision of platform-verifier or webpki-roots is probably only about frequency of updates to webpki-roots needed to keep the service running.

[complexspaces]

@cpu I think your explanation looks great! Thanks for writing that out here and tagging me in.

After reading through this thread, I have a few followup comments:

which means this dep alone is almost a 20% increase in dependencies... (as a side note, someone inspecting Cargo.lock will probably be surprised finding "jni" in their project - even if it's only used on android)

This is an understandable concern! I tend to have a similar mindset myself when looking at dependency trees. However the other factor I consider is versioning: Are all the crates using versions that don't change often (or at all)? This significantly increases the chance downstream consumers lockfiles will merge together nicely and prevent the (worse, IMO) problem of duplicates from appearing. With all of that said, once the new version of security-framework has some time to settle in the ecosystem we could update to it and drop all 3 of the num-* dependencies, leaving basically only crates required to interact with the OS. Its not reasonable for us to maintain all these FFI bindings by hand.

Regarding the usage assumptions:

I can only speculate, but I suspect that means the most dominant environment where ureq runs is Linux (inside a Docker container)

I don't think this is strictly true, IME. I have seen ureq be chosen by a number of command-line tools and "rust scripts" due to its easy API and small compile time impact. So that means its running on a lot of "desktop OSes" like Windows and and macOS too. A quick glance at the reverse dependencies has evidence as well.

the number of disparate certs the ureq instance would be subjected to is very limited.

I think I agree so far on this point: most tooling has an expected set of destinations baked in. But, there remains some value though, like if you are in an environment that forces network access to go through a proxy which requires a cert.

[algesten]

DECISION: Leaving as-is (using platform-verifier by default) for 3.x-rc1. I will keep this PR open since it's still on my mind.

Thank you both @cpu and @complexspaces very much for helping me with this one.

[algesten]

Another thought (and I don't expect anyone to answer this, I'm just capturing my thoughts on this PR)

reqwest is a very good http client. It got a "blocking" feature, which makes it roughly equal to ureq. The history of ureq is that it started as a counter-reaction to needing a lot of dependencies for making a few HTTP calls. It might be true to say that one key differentiating factor in choosing ureq over reqwest, is the dependency count.

If we try to pick the features "blocking", TLS, JSON and GZIP. Then count the deps and try to build.

reqwest:

martin@nugget:~/dev/reqwest$ cargo tree -e normal --no-default-features --features "blocking default-tls json gzip" --prefix none --no-dedupe | sort | uniq | wc -l
      69
martin@nugget:~/dev/reqwest$ cargo clean && time cargo build --no-default-features --features "blocking default-tls json gzip"
...
real	0m6.686s
user	0m31.528s
sys	0m3.094s

ureq with platform-verifier:

martin@nugget:~/dev/ureq$ cargo tree -e normal --no-default-features --features "rustls json gzip" --prefix none --no-dedupe | sort | uniq | wc -l
      57
martin@nugget:~/dev/ureq$ cargo clean && time cargo build --no-default-features --features "rustls json gzip"
...
real	0m7.914s
user	0m32.379s
sys	0m3.336s

ureq without platform-verifier:

martin@nugget:~/dev/ureq$ cargo tree -e normal --no-default-features --features "rustls json gzip" --prefix none --no-dedupe | sort | uniq | wc -l
      48
martin@nugget:~/dev/ureq$ cargo clean && time cargo build --no-default-features --features "rustls json gzip"
...
real	0m7.013s
user	0m26.842s
sys	0m2.942s

It's not very scientific, but is another data point. I should definitely take a closer look at the other deps in ureq.

[algesten]

I've decided to merge this PR, and it's not a decision made lightly (still not sure it's the right one).

ureq main selling points are and always were:

• low dependency count
• sync

With rustls-platform-verifier as the default, ureq's dependency count end up being roughly equivalent to reqwest. In my opinion that means there isn't enough difference to motivate ureq's existence. ureq2 also defaulted to webpki-roots, which means whatever success ureq has had so far, it was with that setup.