-
Notifications
You must be signed in to change notification settings - Fork 3k
throttle npm http requests #7200
Comments
I will give this a try. But I don't think this will scale as I have a lot of developers who would need to use crapify to avoid this encountering this firewall issue. |
My thought was that you could install it sitewide and see if it helps (that's what it was written to do). To make npm throttle its connections in an orderly way would mean two things:
I'm completely sympathetic to your need to come up with a solution to this problem, because all of the companies we've encountered running into this difficulty are large, powerful enterprises we'd very much like to make happy. The trick is finding a way to do it that doesn't require us to put what would probably be at least a month of effort into rewriting one of the core components of the product. |
I am experimenting with crapy. I still don't know if it will be a viable solution for our problem. At the moment it is quite good in helping me debug issues. By eliminating issues that can be caused by concurrency I can focus on other issues that are also happening. As a side note, It would be really nice to be able to enable time stamps in npm logging. This would make using the logs as part of internal support requests easier. |
@aetsher no problem :-) |
crapify solves one problem but leaves me with an other. I need to use a proxy for external calls (ex: postinstall call to github for certain libs). By defining a proxy in the npm config all those calls seem to now go through the crapify proxy and never find github. |
@zenwork that's interesting that the GitHub requests aren't getting proxied properly. Could you share a package.json snippet with me that fails? Hopefully something will jump out at me that I can patch. |
Here is the anonymized package.json I am using. Not that I am using artifactory 3.4.1 as an npmjs cache.
|
I got a piece of information from the firewall team which could help. It seems that too many sessions rather than too many requests are generating all the HTTP 503 from the firewall. Does that make any sense to you? I can't imagine that npm is trying to open an HTTP session with each call. |
If you install a package that does not have git dependencies, does On Tuesday, January 27, 2015, Florian Hehlen notifications@github.com
|
The source has been found. Our firewall was creating an http session for each npm request (creating a cookie). So we were hitting the max sessions/IP limit and causing everything to go very bad. Things are back to normal now that they have removed this 'feature'. thanks for all the help. If nothing else it has been very instructive :-) |
I'm going to close this as resolved (for now?!). Let us know if you run into further issues, @zenwork! |
I've experienced it so many times.. Please re-open this. It's especially painful on El Capitan, it can't handle too many concurrent connections and crashes wi-fi... |
I could confirm the issue. I'm also using El Capitan and every time |
This problem remains and really is a issue. A usual download manager would only allow about three http connections at a time. If you install a fresh set of npm modules from scratch, npm may open > 200 http connections at once, which I've seen block the network connectivity of a complete household several times. This happens reliably and reproducible and is really annoying. Not only this may block npm itself, since it is scheduling some hundred stale connections and not progressing any more. It also blocks other user's internet connections, if npm has used up the maximum number of connections available for that household. This is really a pain. Is there any progress on this issue? Edit: The committers seem to be aware of that problem, but through another ticket: #4553 |
The CLI team has as a high-level roadmap item improving the performance and robustness of npm's networking code, which in all likelihood means replacing |
Same problem here, and I have gigabit Internet... Seriously. You should at last in the meantime add an option to throttle the number of simultaneous connections. |
The CLI has been doing this since |
Thanks for the info! This is very good to know. I was able to make it work by disabling firewall + rebooting router, but if the issue ever happens again I'll give it a try. |
I have posted this info at a few places where people encountered the issue and this option was never mentioned. |
Hi,
Is there a way to throttle npm when calling 'install' or 'update' so that dependency resolution does not create so many concurrent http calls?
npm is causing some trouble with on of our corporate firewall. We have our own internal npm repo and the issue is occurring between the npm client and our repo.
At times too many concurrent requests are going through and causing the firewall to throw HTTP 503 errors back. This does not kill the npm install but results in builds taking 4 to 5 times longer than usual. Sometimes it seems to cause some fatal timeouts or the dreaded 'cb() never called'
I have already tried to get the firewall changed to a more lenient setting but it is not possible as this is a wanted feature on this firewall.
cheers,
Florian
The text was updated successfully, but these errors were encountered: