"cargo install --locked" very slow with sparse-registry

[tshepang]

Problem

cargo install --locked takes over 20s at times when ran in latest ripgrep Git repo, compared to the less than 5s of cargo install.

Version

cargo 1.64.0 (c0bbd42ce 2022-07-03)
release: 1.64.0
commit-hash: c0bbd42ce5e83fe2a93e817c3f9b955492d3130a
commit-date: 2022-07-03
host: x86_64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/3.0.3)
os: Debian testing (bookworm) [64-bit]

[ehuss]

Can you say more about what specifically you are referring to with your timing? Is that the time to download the crates? Or is that the total time to build? Is there a similar time difference without -Zsparse-registry? Are those timings with an empty CARGO_HOME?

For me, running:

CARGO_HOME=chome hyperfine --show-output --runs 3 --prepare "rm -rf chome" "cargo install ripgrep" "cargo install --locked ripgrep" "cargo install ripgrep -Zsparse-registry" "cargo install --locked ripgrep -Zsparse-registry"

shows:

Summary
  'cargo install ripgrep -Zsparse-registry' ran
    1.01 ± 0.04 times faster than 'cargo install --locked ripgrep -Zsparse-registry'
    3.44 ± 0.09 times faster than 'cargo install --locked ripgrep'
    3.44 ± 0.07 times faster than 'cargo install ripgrep'

So the locked/unlocked times are nearly identical.

[tshepang]

Am running from inside ripgrep repo...

❯ hyperfine --show-output --runs 3 --warmup 1 \
    "cargo install --path ." \
    "cargo install --locked --path ." \
    "cargo install --path . -Zsparse-registry" \
    "cargo install --locked --path . -Zsparse-registry"
...
Summary
  'cargo install --path . -Zsparse-registry' ran
    3.34 ± 0.19 times faster than 'cargo install --path .'
    3.39 ± 0.17 times faster than 'cargo install --locked --path .'
   79.84 ± 102.30 times faster than 'cargo install --locked --path . -Zsparse-registry'

[tshepang]

A smaller test...

hyperfine --show-output --warmup 1 --runs 1 \
      "cargo install --path . -Z sparse-registry" \
      "cargo install --path . -Z sparse-registry --locked"
...
Summary
  'cargo install --path . -Z sparse-registry' ran
  107.17 times faster than 'cargo install --path . -Z sparse-registry --locked'

The main delay is at Fetch stage...

Fetch [=========================> ] 33/34

[weihanglo]

CARGO_LOG=cargo::ops::registry=debug CARGO_HTTP_DEBUG=true cargo +nightly install --path . -Z sparse-registry --locked

The slow result is not always reproducible, but I found that if it finishes in a fast speed, the x-cache header shows "Hit from cloudfront". Otherwise "RefreshHit from cloudfront". I'll back then to investigate more.

• http-debug: < x-cache: Hit from cloudfront
• http-debug: < x-cache: RefreshHit from cloudfront

[ehuss]

Hm, I'm able to reproduce this now 100% of the time.

There is a lot of complexity here that is a bit tricky to untangle. With --locked, cargo install does a separate resolution step to check for yanked dependencies. I'm not sure why that would change the behavior so substantially. From what I can tell, cargo is making a request for every dependency, but with --locked it seems to not issue them in parallel. I think this will take quite a bit more investigation. @arlosi is this something you can look at?

[arlosi]

I can investigate.

@rustbot claim