Cargo stuck for hours, taking 100% of the CPU #4604
Assignees
Comments
|
I don't know if it's easily reproducible, but |
|
By freeze, I mean, get stuck on |
|
After few hours (I think 3 or 4 hours), this is what the Cargo.lock looks like: https://gist.github.com/paulrouget/385dd3d86028c698c98776d8d198e61a The difference is the move from a local checkout of servo, webrender and webrender_api to the github repo. |
|
This is likely an unfortunate instance of #4066, although I'd have to dig in to be sure |
In this issue, you say:
But in this case, it works. But it takes 4 hours. |
|
Is there any workaround? Because every time I need to update servo in my projects, I have to run Cargo for hours before being able to do anything. |
|
I don't think I understand what's going on enough to recommend a workaround or know how to fix it. If it's not #4066 then Cargo may just need a tweak to a heuristic or something like that. I do not have time to investigate right now, but I can try to get around to this soon. |
|
Could this be a slow libgit operation? That is, #4261 ? |
|
I tried to reproduce today with: #4604 (comment) But suddenly it's fast. What changed since last week:
|
|
This is still happening. Stuck on: |
|
Would it be possible to work around the issue by pulling manually? I checkout-ed commit #43a4f01 in /Users/paul/.cargo/git/checkouts/servo-12bfca331f128de3/43a4f01/ - but still, it gets stuck exactly the same way. |
|
I attached lldb to the process and dumped the backtrace: https://gist.github.com/paulrouget/d6e3d62c5fd03af96b3338cf9414335d |
|
@paulrouget it looks like it's stuck in libgit2 fetching? If that's the case I'm not sure how to debug... You could try out #4646 locally and perhaps see if progress is being made? |
Now this has landed, I see that it's progressing, but very very slowly. It's apparently stagnating at 1 minute per percent. The network monitor tells me it's downloading at 2 to 7 kb/s. |
|
I can slow down a lot more. At the moment, it's around 7 minutes per percent. |
|
Cargo runs at 100% CPU: |
|
Most of time spent in this if I read the samples right: |
|
Ok well at least we know it's stuck in libgit2! Do you know the URL of what it's cloning? If you can create a standalone program using just git2-rs that clones it will probably be sufficient for opening a bug against libgit2 |
|
Does Cargo use https://github.com/alexcrichton/git2-rs? If so, |
|
Yes Cargo uses git2-rs, but Cargo likely doesn't match the exact behavior of one of the examples, but rather something may need to be extracted from Cargo's source to match its patterns. |
|
I reduced this problem to the most minimal test case I could. I think it might have to do with disk-to-disk operations. Not sure. See https://github.com/alexcrichton/git2-rs/issues/280 |
alexcrichton
added a commit
to alexcrichton/cargo
that referenced
this issue
Jan 8, 2018
This commit updates the handling of git checkouts from the database to use hardlinks if possible, speeding up this operation for large repositories significantly. As a refresher, Cargo caches git repositories in a few locations to speed up local usage of git repositories. Cargo has a "database" folder which is a bare checkout of any git repository Cargo has cached historically. This database folder contains effectively a bunch of databases for remote repos that are updated periodically. When actually building a crate Cargo will clone this database into a different location, the checkouts folder. Each rev we build (ever) is cached in the checkouts folder. This means that once a checkout directory is created it's frozen for all of time. This latter step is what this commit is optimizing. When checking out the database onto the local filesystem at a particular revision. Previously we were instructing libgit2 to fall back to a "git aware" transport which was exceedingly slow on some systems for filesystem-to-filesystem transfers. This optimization (we just forgot to turn it on in libgit2) is a longstanding one and should speed this up significantly! Closes rust-lang#4604
bors
added a commit
that referenced
this issue
Jan 8, 2018
Leverage local links on git checkouts This commit updates the handling of git checkouts from the database to use hardlinks if possible, speeding up this operation for large repositories significantly. As a refresher, Cargo caches git repositories in a few locations to speed up local usage of git repositories. Cargo has a "database" folder which is a bare checkout of any git repository Cargo has cached historically. This database folder contains effectively a bunch of databases for remote repos that are updated periodically. When actually building a crate Cargo will clone this database into a different location, the checkouts folder. Each rev we build (ever) is cached in the checkouts folder. This means that once a checkout directory is created it's frozen for all of time. This latter step is what this commit is optimizing. When checking out the database onto the local filesystem at a particular revision. Previously we were instructing libgit2 to fall back to a "git aware" transport which was exceedingly slow on some systems for filesystem-to-filesystem transfers. This optimization (we just forgot to turn it on in libgit2) is a longstanding one and should speed this up significantly! Closes #4604
|
Thank you so much! This is going to make my life better :) |
|
It'll take a bit to get this into nightlies btw, I've opened a PR at rust-lang/rust#47280 but it may take some time to land. |
|
The same issue with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
While working on Servo, on a regular basis, Cargo gets stuck on
Updating git repositoryorUpdating registry.I usually have to wait few hours until it finishes. Sometimes with no error (compilation went through). Sometimes with an error (conflict of version with a dependency in my Cargo.toml and Servo's Cargo.toml).
I guess #4502 is related.
Is there anything I can do to help diagnose this problem?
The text was updated successfully, but these errors were encountered: