Cargo publish fails with 'Too many open files' #4403
Labels
Comments
|
Oh dear, this is definitely not expected! I'm not really sure what's going on here. Would it be possible to get stack traces of Cargo when it starts opening files it's not closing? I'm curoius what API we're calling in libgit2 that causes this... |
|
This is similar to https://github.com/onur/docs.rs/issues/76. I've temporary solved this problem with running |
carols10cents
added
A-git
|
This is probably caused by libgit2/libgit2#2758 |
|
If we want to fix this, which I think we should, I think it'd look something like -- On the "slow path" of doing a git fetch, just before we do that we should check how many pack files are in the repository. Above some threshold we "do a gc" by blowing away the entire index. That means that sporadically an update could take awhile but it never would on the "fast path" and otherwise it'd help keep the repo in a relatively pristine state. |
|
Further discussion on #libgit2 --
|
alexcrichton
added a commit
to alexcrichton/cargo
that referenced
this issue
Sep 11, 2017
This commit is targeted at improving the long-term management of git checkouts and git repositories. Currently every time data is fetched from crates.io libgit2 will create a new pack file in the repository. These pack files accumulate over time and end up causing pathological behavior if there's lots of them, causing libgit2 to open many file descriptors all at once, possibly blowing the system's file descriptor limits. To alleviate this problem you typically run `git gc`, but libgit2 doesn't have this implemented. Instead what Cargo now does is detect this situation and run literally the command line tool `git gc` in a best-effort attempt to compact the repo. Failing that, for example when git isn't installed, Cargo will remove the entire repo and do a full checkout again. At the same time this commit also generalizes this logic, plus the existing fast path github logic, to all git repositories and not just the index. That way all git repositories can benefit from the "github fast path" as well as the compaction steps. Closes rust-lang#4403
bors
added a commit
that referenced
this issue
Sep 11, 2017
Periodically gc repos in Cargo This commit is targeted at improving the long-term management of git checkouts and git repositories. Currently every time data is fetched from crates.io libgit2 will create a new pack file in the repository. These pack files accumulate over time and end up causing pathological behavior if there's lots of them, causing libgit2 to open many file descriptors all at once, possibly blowing the system's file descriptor limits. To alleviate this problem you typically run `git gc`, but libgit2 doesn't have this implemented. Instead what Cargo now does is detect this situation and run literally the command line tool `git gc` in a best-effort attempt to compact the repo. Failing that, for example when git isn't installed, Cargo will remove the entire repo and do a full checkout again. At the same time this commit also generalizes this logic, plus the existing fast path github logic, to all git repositories and not just the index. That way all git repositories can benefit from the "github fast path" as well as the compaction steps. Closes #4403
alexcrichton
added a commit
to alexcrichton/cargo
that referenced
this issue
Sep 14, 2017
This commit is targeted at improving the long-term management of git checkouts and git repositories. Currently every time data is fetched from crates.io libgit2 will create a new pack file in the repository. These pack files accumulate over time and end up causing pathological behavior if there's lots of them, causing libgit2 to open many file descriptors all at once, possibly blowing the system's file descriptor limits. To alleviate this problem you typically run `git gc`, but libgit2 doesn't have this implemented. Instead what Cargo now does is detect this situation and run literally the command line tool `git gc` in a best-effort attempt to compact the repo. Failing that, for example when git isn't installed, Cargo will remove the entire repo and do a full checkout again. At the same time this commit also generalizes this logic, plus the existing fast path github logic, to all git repositories and not just the index. That way all git repositories can benefit from the "github fast path" as well as the compaction steps. Closes rust-lang#4403
bors
added a commit
that referenced
this issue
Sep 14, 2017
Periodically gc repos in Cargo This commit is targeted at improving the long-term management of git checkouts and git repositories. Currently every time data is fetched from crates.io libgit2 will create a new pack file in the repository. These pack files accumulate over time and end up causing pathological behavior if there's lots of them, causing libgit2 to open many file descriptors all at once, possibly blowing the system's file descriptor limits. To alleviate this problem you typically run `git gc`, but libgit2 doesn't have this implemented. Instead what Cargo now does is detect this situation and run literally the command line tool `git gc` in a best-effort attempt to compact the repo. Failing that, for example when git isn't installed, Cargo will remove the entire repo and do a full checkout again. At the same time this commit also generalizes this logic, plus the existing fast path github logic, to all git repositories and not just the index. That way all git repositories can benefit from the "github fast path" as well as the compaction steps. Closes #4403
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
While trying to publish a crate to a local instance of crates.io, cargo fails with a 'Too many open files' error while creating its lock file:
To put things into context, I have approximatively a thousand version of a single crate published in my local registry (maybe not a legit use-case per se, but I still think the problem is worth an issue).
By running the program with
strace, we can see that cargo tries to open every git pack file in the registry:One working wordaround is updating the file descriptor limit with
ulimitto more than 1024.I'm not familiar enough with cargo to know why it would need to have all these git pack files open at the same time. Is this an expected behaviour of cargo ? Would this happen with 1 published version for 1000 crates ?
Complete
strace -e open,close -f'd cargo log:https://gist.github.com/kureuil/989b1f4c6d283384c8b70fdd1ddf2e87
The text was updated successfully, but these errors were encountered: