[BUG] Getting constant Client network socket disconnected before secure TLS connection was established
#4652
Comments
hanzlamateen
added
Bug
|
Same issue here. Started seeing it after upgrading to Node 16.14.x. The URL reported in the ECONNRESET failure message is random (changes). I see this on MacOS as well as Linux (docker image). In case it helps someone, there are two potential workarounds (the second one being sketchy).
I would think this is being experienced by many people and therefore would request that be looked at urgently. |
|
A couple of months ago we started seeing network errors as well, mostly Recent npm upgrades received via Docker Node.js base images are a probable cause, because our infrastructure hasn't changed significantly. It started in February with only 1-2 scheduled builds per week failing, now 9/10 builds are failing with network errors. I'm not 100% sure, but I feel that |
|
if all of you who are experiencing this error are proxy users, then i have a good idea of what's going on. is anyone seeing this not using a proxy? |
I am not using any proxy |
|
I have not seen this issue even once when not using a proxy. By not using a proxy I mean, "proxy" and "https-proxy" are not set in the npm config, and importantly, HTTP_PROXY/HTTPS_PROXY/http_proxy/https_proxy variables are not set in the system environment. |
|
can confirm having had that issue for a while now, not using any proxy, and it seems to be sometimes fixed by setting the NO_PROXY env so e.g. so before_script: or before_script: but not every time ... or I just was lucky with 6 jobs this morning and now a 7th is failing always with the usual error: npm ERR! code ECONNRESET |
|
Just found this issue - I posted to #3078 (comment) although that issue is closed so perhaps here is better. I'm seeing this issue without a proxy. More detail in the post above. |
|
I re-ran things, this time using I decided to leave a PCAP running on my edge router during this attempt. The PCAP was taken on the LAN side just so I would filter on the originating IP address of the Raspberry Pi. The initial connection started just fine. We see DNS queries for npm makes a single TLS connection to the first host on the list, namely e.g:
After about 5 mins, there are multiple other TLS connection attempts to the same host using different sockets. Each of these attempts is refused and rest ( Soon after, we then see a bunch more DNS requests for the same thing (and answers):
As then, the Pi (npm) opens multiple TLS connections in a "second wave" to all these hosts (based on the DNS answers):
And even multiple TLS connections to the same host. e.g. filter on
Concentrating on just this host (
However, other sockets are opened and TLS connections are attempted but are then reset by the host:
This pattern seems to be similar to all the hosts where connections were made during the "second wave". The initial good TLS connection to
I'm not sure if these data connections and management within npm, or on the npm registry host servers, is by design or not. |
|
FWIW, I finally managed to get I decided to reduce the number of concurrent sockets used by the npm client via the Maybe this helps someone, and maybe it helps the devs figure out why the TLS connections keep getting reset by the registry server on TLS Client Hello. |
|
I had the same problem, tried both Using npm 7.24.0 |
|
Somehow the situation improved greatly in the last couple of weeks in my case. I have had only a single build breaking due to "network problems" (ECONNRESET). I suspect newer versions of npm have had some sort of fix for this issue…? |
|
We're seeing a similar issue happening in our environment, as well, and from the symptoms I assume this is the same issue as described here. Some more context, as it may help to investigate:
After upgrading nodejs to 16 with npm 8.11.x we see builds failing (up to 90% of the times, but at least 50% of the times) with the mentioned error message. To get more visibility into this issue, I setup an intermediate proxy using tinyproxy, so that the connection flow looks like this: After running As a result, however, the builds with having tinyproxy as a proxy between npm and the upstream proxy, succeeds more often (needless to say that this takes a significant amount of additional time, as connections need to be retried after the current 100 connections drained). Our upstream proxy does not have a quota (apart from a hard limit of connections the proxy can open) for concurrent connections. This might indicate, that the issue is related to the number of connections opened and held open by npm or the underlying network stack? Sadly I do not have enough knowledge of the underlying code of npm and node to make such an assumption of even prove that it is right :( I'm a bit out of ideas on what the issue here might be. So far, we did not see this issue either without a proxy or with npm 6. If I can provide any more information that might be of help to investigate this issue, please let me know. |
|
We further investigated the issue. In our case, when using a downstream proxy which supports https for the connection from npm to the first proxy, we don't see this issue happening, anymore. We used mitmproxy (mitmdump) which results in the following network chain: Running After that, I traced the packets on the machine where From my not so deep understanding, this looks complicated in a way that the connection seems to be closed by the proxy while the client wants to send an encrypted alert (which contents we're unfortunately not aware of). In a trace with npm 6 this is, with the same proxy setup, this is not happening. In this case, the connection is correctly closed with a FIN, FIN ACK flow, as one would expect. It hink there are two issues here: The proxy server, who is rejecting any more traffic after the connection was closed from their side, however, I'm not sure why npm is behaving so differently in these two versions. With this behaviour in mind, I searched a bit and were able to find issue nodejs/node#23128 in nodejs. I'm not sure if that is more related to the issue we are seeing or not, just wanted to mention it. The issue is pretty old as well. This comment will probably confuse more than it really helps, however, I wanted to give our findings here in this issue as well :) |
|
Had the same problem with npm install in docker lst-alpine (3.16). Btw in osX everything was ok. Using npm 8.11.0 |
|
also getting this issue on
no proxy, sending a curl to the relevant requests works fine. i did find a solution for my needs, which is just downgrading node to debug log0 verbose cli C:\Program Files\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js
1 info using npm@9.6.4
2 info using node@v20.0.0
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 1ms
5 timing config:load:file:C:\Users\{user}\AppData\Roaming\nvm\v20.0.0\node_modules\npm\npmrc Completed in 0ms
6 timing config:load:builtin Completed in 0ms
7 timing config:load:cli Completed in 1ms
8 timing config:load:env Completed in 1ms
9 timing config:load:project Completed in 0ms
10 timing config:load:file:C:\Users\{user}\.npmrc Completed in 1ms
11 timing config:load:user Completed in 1ms
12 timing config:load:file:C:\Program Files\nodejs\etc\npmrc Completed in 0ms
13 timing config:load:global Completed in 0ms
14 timing config:load:setEnvs Completed in 1ms
15 timing config:load Completed in 5ms
16 timing npm:load:configload Completed in 5ms
17 timing npm:load:mkdirpcache Completed in 0ms
18 timing npm:load:mkdirplogs Completed in 0ms
19 verbose title npm i yarn
20 verbose argv "i" "--global" "yarn"
21 timing npm:load:setTitle Completed in 1ms
22 timing config:load:flatten Completed in 2ms
23 timing npm:load:display Completed in 2ms
24 verbose logfile logs-max:10 dir:C:\Users\{user}\AppData\Local\npm-cache\_logs\2023-04-19T19_37_33_000Z-
25 verbose logfile C:\Users\{user}\AppData\Local\npm-cache\_logs\2023-04-19T19_37_33_000Z-debug-0.log
26 timing npm:load:logFile Completed in 6ms
27 timing npm:load:timers Completed in 0ms
28 timing npm:load:configScope Completed in 0ms
29 timing npm:load Completed in 15ms
30 timing config:load:flatten Completed in 0ms
31 timing arborist:ctor Completed in 0ms
32 silly logfile start cleaning logs, removing 2 files
33 timing idealTree:init Completed in 8ms
34 timing idealTree:userRequests Completed in 3ms
35 silly idealTree buildDeps
36 silly fetch manifest yarn@*
37 silly logfile done cleaning log files
38 silly placeDep ROOT yarn@ OK for: want: *
39 timing idealTree:#root Completed in 88956ms
40 timing idealTree:node_modules/yarn Completed in 0ms
41 timing idealTree:buildDeps Completed in 88957ms
42 timing idealTree:fixDepFlags Completed in 0ms
43 timing idealTree Completed in 88968ms
44 timing command:i Completed in 88975ms
45 verbose type system
46 verbose stack FetchError: request to https://registry.npmjs.org/yarn failed, reason: Socket connection timeout
46 verbose stack at ClientRequest. (C:\Users\{user}\AppData\Roaming\nvm\v20.0.0\node_modules\npm\node_modules\minipass-fetch\lib\index.js:130:14)
46 verbose stack at ClientRequest.emit (node:events:511:28)
46 verbose stack at TLSSocket.socketErrorListener (node:_http_client:495:9)
46 verbose stack at TLSSocket.emit (node:events:523:35)
46 verbose stack at emitErrorNT (node:internal/streams/destroy:151:8)
46 verbose stack at emitErrorCloseNT (node:internal/streams/destroy:116:3)
46 verbose stack at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
47 verbose cwd E:\projects\{project}\{branch}
48 verbose Windows_NT 10.0.22621
49 verbose node v20.0.0
50 verbose npm v9.6.4
51 error code ERR_SOCKET_CONNECTION_TIMEOUT
52 error errno ERR_SOCKET_CONNECTION_TIMEOUT
53 error request to https://registry.npmjs.org/yarn failed, reason: Socket connection timeout
54 verbose exit 1
55 timing npm Completed in 89007ms
56 verbose unfinished npm timer reify 1681933053032
57 verbose unfinished npm timer reify:loadTrees 1681933053035
58 verbose code 1
59 error A complete log of this run can be found in: C:\Users\{user}\AppData\Local\npm-cache\_logs\2023-04-19T19_37_33_000Z-debug-0.log
|
|
nevermind, the solution is to cancel your service with spectrum/charter. turned out i couldn't ping the ipv6 route for lucky for me a competitor finally became available in my area though it would be nice if npm used ipv4 as fallback if ipv6 fails |
|
I closed all of the ports (except for 22, 80 and 443) on my Ubuntu server and npm stopped working. Thank you @swinster for your WireShark screenshots which showed that npm uses 59376 port. Opening it on my server helped and npm started working again. |
|
I am using nodejs 18.19.1 npm 10.2.4 and also getting this network socket error. It is reproduced on every clean install. If you retry then install is ok. |
|
Updating Docker Desktop to 4.29.0 fixed the issue for me. I didn't need --max-sockets |
|
This is still an issue, in the core. The NPM dev team should have a look at it with all the information above. I encounter this issue within for example on an Azure Container Registry build agent with a QEMU linux/arm64 docker buildx image build. I tried the following approaches:
|
Is there an existing issue for this?
This issue exists in the latest npm version
Current Behavior
When trying to install packages using
npm install --production=false --loglevel notice --legacy-peer-deps, we get constantly getting mentioned error. The package that fails is always different, so there isn't a specific package causing the issue.Here are the changes I have tried so far:
Added
npm install -g npm, still not workingAdded following before
npm i, still not workingnpm config set fetch-retry-maxtimeout 9999999 -gUpdated
npm icommand to include timeout, still not workingnpm install --production=false --loglevel notice --legacy-peer-deps --timeout=9999999Installed VPN and connected to other regions, still not working.
Expected Behavior
Install packages without failing
Steps To Reproduce
Using a project with workspaces
No cache available
npm install --production=false --loglevel notice --legacy-peer-deps
Environment
The text was updated successfully, but these errors were encountered: