"Too many open files" w/ Async Client #386
Comments
Hm, possibly. There'd need to be some way to notice that some file descriptors have been closed (and the amount is shared with the whole process, not just what reqwest uses). In your example, it doesn't look like that many connections are being opened, can you increase the |
|
I'm a little confused myself, since this happens on a fresh start with just that application running. |
Possibly wrong error message? |
|
The limit is per-process, so other applications wouldn't affect it. You can check it via |
|
Claims the max open files to be 1024, though I don't think that many files are being opened by reqwest. fn get_request<'a>(client: &'a Client, url: String) -> impl Future<Item = Response, Error = DistUpdateError> + 'a {
client.get(&url)
.send()
.inspect(move |_v| {
eprintln!("GET {}", url);
})
.and_then(|resp| resp.error_for_status())
.map_err(|why| DistUpdateError::Request { why })
} |
|
According to |
|
Ah, nevermind, I see that this occurred: |
|
It seems to be continually creating new |
|
It calls this repeatedly until it eventually runs out of fds: |
|
Also occurs when using a |
|
Oh no, that's not good! Could you share what versions of reqwest and hyper are listed in your If you don't mind, grabbing the logs from |
seanmonstar
added
C-bug
|
That would be Included two trace logs[0]: one with just the trace logging, and one with both that and strace output. [0] logs.zip |
|
May also want to know that this is being run in a |
|
Could you try pinning hyper to 0.12.13? I don't have a cause yet, but there was some changes to the connection pool in the 0.12.14 release... |
|
Same issue with |
|
Sorry for taking a little while. I've read through the logs, and it actually seems like the huge increase in connections happens at a specific point in the app. As if something is suddenly spawning >1000 requests on the client. The client doesn't currently put a limit on the number of outstanding requests or connections it can manage, that's left up to the user. You can use something like |
seanmonstar
removed
C-bug
|
same issue with reqwest 0.8.8 after cargo update, the fd grow up all the time until crash... |
|
As mention in my previous comment, you'll need to limit the number of requests you make at a time, or increase the FD limit in your OS. |
|
Thank you for a great library! Just a head's up, in case it helps anybody: We use One user ran into a problem with We're going to try downgrading to |
|
OK, as of this morning, it appears that reverting to We use /// Fetch an existing resource.
pub fn fetch<R: Resource>(&self, resource: &Id<R>) -> Result<R> {
let url = self.url(resource.as_str());
let client = reqwest::Client::new();
let res = client.get(url.clone())
.send()
.map_err(|e| Error::could_not_access_url(&url, e))?;
self.handle_response(res)
.map_err(|e| Error::could_not_access_url(&url, e))
}We're not directly using Is there something we can do to help troubleshoot this? |
|
@emk yikes! Are you actively making thousands of requests simultaneously? Or does it seem like the connections are just idling? I noticed you're using reqwest 0.8, which its dependencies don't really see any more updates. I believe there were some bugs in hyper that would leave connections open, fixed in newer versions. Also, if the host is often the same, you could reuse connections and also better performance by sharing a |
|
@seanmonstar We should have less than 10 I'll see if we can upgrade Thank you for your help! |
* else, runaway async http requests flood the system's capacity for open sockets and get a "too many files open" error (see: seanmonstar/reqwest#386 (comment)) * as remedy: use a buffered stream (TODO: benchmark to find best concurrency limit)
I've been developing the tooling for an
aptreplacement with the async client, which may need to fetch a couple dozen files at the same time (depending on how many sources are active and what files those sources' release files point to).The following error occurs when sharing a single
async::Clientacross all connections:Is this something that could be handled better by reqwest? Perhaps return a
NotReadywhen there's too many files open.The text was updated successfully, but these errors were encountered: