Updating crates.io index sometimes takes very long

[e00E]

When building a project with cargo, sometimes cargo starts with "updating crates.io index". Sometimes this process takes very long. Today I observed this this step displaying "resolving deltas" for maybe 5 minutes.

This is surprising because getting the index from scratch after deleting is ~/.cargo/registry is faster than updating the existing index. (I'm assuming this cargo step doesn't touch other directories.) If doing it from scratch is faster, than why isn't cargo doing that?

I observe this from time to time but it happens rarely enough that I didn't bother reporting it until now. I do not know how to reproduce it reliably. I have also observed this happening to other people new to Rust and cargo because this step takes so long for what feels like no reason. This was frustrating for them.

There are probably good reasons for the current behavior. I still feel this is worthy of a bug report. Maybe there could be heuristics to determine when getting the index from scratch would be faster.

Here is a description of my setup in case it helps debug the performance problem. I use cargo daily on projects with many dependencies. My machine runs Linux, has an SSD, 8 core / 16 thread recent processor, 32 GB RAM. My internet download speed is 100 Mbit/s. Nothing else significant was running when cargo was slow.

Version

cargo 1.66.0 (d65d197ad 2022-11-15)
release: 1.66.0
commit-hash: d65d197ad5c6c09234369f219f943e291d4f04b9
commit-date: 2022-11-15
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.15.0 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q)
os: Arch Linux Rolling Release [64-bit]

[Eh2406]

This is a known long-standing bug with libgit2. The process of a clean clone of a repository is linear in the history, but the process of updating into an existing repository is quadratic. Also cargoes index is big and can have a lot of history. We regularly squash the history to try and reduce the problem. sparse-registry will hopefully be stabilized soon removing git from downloading the index at all. There is also work to switch from libgit2 to gitoxide which would also avoid this problem.

[Eh2406]

I'm going to consolidate in #11014

[e00E]

Thanks for the explanation and linking to the other issue. I could have sworn I searched for open "resolving deltas" issues and it didn't find that one.