Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to connect to establish a TCP connection (ECONNREFUSED) #40702

Closed
JeffreyArt1 opened this issue Nov 2, 2021 · 38 comments
Closed

Unable to connect to establish a TCP connection (ECONNREFUSED) #40702

JeffreyArt1 opened this issue Nov 2, 2021 · 38 comments
Labels
net Issues and PRs related to the net subsystem.

Comments

@JeffreyArt1
Copy link

JeffreyArt1 commented Nov 2, 2021

Version

v17.0.1

Platform

Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

Subsystem

No response

What steps will reproduce the bug?

I'm trying to connect to a micro-service with this piece of code:

const connect = (host, port) => {
  socket = new require('net').Socket();
  socket.connect(port, host)
    .on('connect', () => console.log('konected!'))
    .on('error', e => console.log(e));
};

connect('localhost', 5098);

this prints out:

Error: connect ECONNREFUSED ::1:5098
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -61,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '::1',
  port: 5098
}

How often does it reproduce? Is there a required condition?

Since installed node 17

What is the expected behavior?

No errors.

What do you see instead?

Error: connect ECONNREFUSED ::1:5098
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16) {
  errno: -61,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '::1',
  port: 5098
}

Additional information

As far a i tested (v14.18.1 and v16.12.0), this only happens in this version of node.

@Mesteery Mesteery added the net Issues and PRs related to the net subsystem. label Nov 2, 2021
@nsainaney
Copy link

nsainaney commented Nov 2, 2021

We have been having the same issue with node 17 and it appears to be a breaking change with how DNS.lookup works. With node 16, the lookup would return a IPv4 address but with node 17, it returns an IPv6 address which will break most REST clients that hardcode URLS like http://localhost:4040/api if the upstream server only binds to the IPv4 address (e.g. server.listen('127.0.0.1'...) etc..

sudo n latest
   installed : v17.0.1 (with npm 8.1.0)
╭─[prompt redacted]›
╰─$ node
Welcome to Node.js v17.0.1.
Type ".help" for more information.
> require('dns').lookup('localhost', null, (e,a,f) => console.log(e,a,f))
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 0,
  hostname: 'localhost',
  oncomplete: [Function: onlookup]
}
> null ::1 6
.exitit
╭─[prompt redacted]›
╰─$ sudo n 16.2.0
   installed : v16.2.0 (with npm 7.13.0)
╭─[prompt redacted]›
╰─$ node
Welcome to Node.js v16.2.0.
Type ".help" for more information.
> require('dns').lookup('localhost', null, (e,a,f) => console.log(e,a,f))
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 0,
  hostname: 'localhost',
  oncomplete: [Function: onlookup]
}
> null 127.0.0.1 4

Notice the results are null ::1 6 vs null 127.0.0.1 4 with null options

@richardlau
Copy link
Member

For reference the change was #39987 -- Node.js no longer re-sorts results of IP address lookups and returns them as-is (i.e. it no longer ignores how your OS has been configured). You can change the behaviour via the verbatim option to dns.lookup() or set the --dns-result-order command line option to change the default.

@nsainaney
Copy link

nsainaney commented Nov 2, 2021

Thanks, @richardlau. How is one supposed to fix this if they are not the author of the server/client. Libraries like axios do not let you pass in the verbatim option so I suppose we all have to use the command line option until packages catch up.

@nsainaney
Copy link

nsainaney commented Nov 2, 2021

Update to my comment above. When dealing with 3rd party libraries, we can use dns.setDefaultResultOrder('ipv4first') and have to be careful to use this in child_processes as well

@wibobm
Copy link

wibobm commented Nov 11, 2021

v17.0.1 and v17.1.0 have same issue with remote debugging

Error in debuggerConnector: connect ECONNREFUSED ::1:49629
Error: connect ECONNREFUSED ::1:49629
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)

@treysis
Copy link
Contributor

treysis commented Nov 12, 2021

@richardlau The better option would of course be to change localhost to 127.0.0.1 when connecting to legacy localhost.

@nsainaney If you cannot fix server/client, the solution would be to stay on the support NodeJS version, i.e. <17, until the authors of server/client add compatibility for Node v17.

@nsainaney
Copy link

Thanks @treysis. There are enough workarounds that this isn't an issue for us.

@treysis
Copy link
Contributor

treysis commented Nov 12, 2021

@nsainaney Yes, I'm just mentioning it because I consider it to be the cleanest way. If you set the setDefaultResultOrder option you might forget about that in the future and will limit IPv6 capability of your applications.

@nsainaney
Copy link

That's a good point. Thank you

timvandermeij added a commit to timvandermeij/pdf.js that referenced this issue Nov 28, 2021
Node.js 17, which as of writing is the most recent version, contains a
breaking change in its DNS resolver, causing Firefox not to start
anymore in our test framework. The inline comment together with the
following resources provide more background:

- nodejs/node#40702
- nodejs/node#39987
- cyrus-and/chrome-remote-interface#467
- https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V17.md#other-notable-changes
- DeviceFarmer/adbkit#209
- https://nodejs.org/api/dns.html#dnssetdefaultresultorderorder

This commit ensures that versions both older and newer than Node.js 17
work as expected. This is mainly necessary since the bots as of writing
run Node.js 14.17.0 which is from before this API got introduced and for
example Node.js 12 LTS is only end-of-life in April 2022, so we have to
keep support for those older versions unfortunately.
Kafva pushed a commit to Kafva/pdf.js that referenced this issue Jan 20, 2022
Node.js 17, which as of writing is the most recent version, contains a
breaking change in its DNS resolver, causing Firefox not to start
anymore in our test framework. The inline comment together with the
following resources provide more background:

- nodejs/node#40702
- nodejs/node#39987
- cyrus-and/chrome-remote-interface#467
- https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V17.md#other-notable-changes
- DeviceFarmer/adbkit#209
- https://nodejs.org/api/dns.html#dnssetdefaultresultorderorder

This commit ensures that versions both older and newer than Node.js 17
work as expected. This is mainly necessary since the bots as of writing
run Node.js 14.17.0 which is from before this API got introduced and for
example Node.js 12 LTS is only end-of-life in April 2022, so we have to
keep support for those older versions unfortunately.
@lydell
Copy link

lydell commented Apr 14, 2022

I run this:

require("http").get("http://localhost:3004/", res => console.log(res.statusCode))
  • In Node.js 16 it prints a status code.
  • In Node.js 17 I get ECONNREFUSED.

Where can I find the OS configuration that Node.js now honors…

  • …on macOS?
  • …on Linux?

Sorry if it feels like a stupid question or like a question that belongs somewhere else, but I think more people than me don’t know this DNS stuff very well and will wonder why things “aren’t working”.

@lexicalunit
Copy link

I run this:

require("http").get("http://localhost:3004/", res => console.log(res.statusCode))

You have to change this to:

const dns = require('dns');
dns.setDefaultResultOrder('ipv4first');
require("http").get("http://localhost:3004/", res => console.log(res.statusCode))

Which should work in both Node v16 and v17, as well as on macOS and Linux.

@lydell
Copy link

lydell commented Apr 14, 2022

@lexicalunit Yes, I read that in the various threads, but that’s not what I’m asking for.

@treysis
Copy link
Contributor

treysis commented Apr 14, 2022

It's difficult to find because bc iirc it depends on sooo many parameters: IPv6 connectivity available? Glibc? Another libc implementation? Systemd?

Instead of changing the dns sorting order globally for a package, it's advised to use 127.0.0.1 when addressing localhost though.

@lydell
Copy link

lydell commented Apr 15, 2022

I don’t quite buy that explanation. If visiting http://localhost:3004 in a browser works and curl http://localhost:3004 works, but require("http").get("http://localhost:3004/", res => console.log(res.statusCode)) fails then it’s gonna look like Node.js is broken. If the explanation is “Well actually, Node.js is correct because it respects hidden OS configuration, just trust me” that doesn’t feel very reassuring?

Note that I’m looking for the default configuration on macOS for example. I bet many developers don’t change anything (knowingly). I’ve tried to find more info myself, but I don’t know what to search for.

@treysis
Copy link
Contributor

treysis commented Apr 15, 2022

Browsers and curl support happy eyeballs, so they try both IPv4 and IPv6. Node has no support (yet?) for happy eyeballs.

DNS lookup in general is documented well. However, localhost is a special case as it's not resolved by the DNS server but before DNS is even asked. It seems to also depend on macOS version. Even for Linux it's not well because again it depends on the library that is being used. So you'd have to look up what the library is doing, and it's also different from distro to distro. E.g.:
NixOS/nixpkgs#19148

Glibc for example also apparently violates some RFC and in some special case maps ::1 to 127.0.0.1, see:
https://sudonull.com/post/60933-Resolve-IP-Addresses-on-Linux-An-Intuitive-and-Detailed-Description

For the time being: don't use localhost but instead use 127.0.0.1.

@lydell
Copy link

lydell commented Apr 20, 2022

Thank you!

I think I understand this better now. I’ll summarize what I (think I have) learned in case it helps others.

When I do require("http").get("http://localhost:3004/"), Node.js needs to turn localhost into an IP address. The default /etc/hosts on macOS starts with:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1	localhost

So if you try to look up localhost in there, you will get two answers: 127.0.0.1 and ::1. So I’ve asked the computer to connect to “localhost”, but now there’s ambiguity.

Node.js 16 and older arbitrarily implemented code that favored ipv4 (127.0.0.1).

Node.js 17 seems to follow the principle of least astonishment, and take the first result from what was returned by the OS when doing the lookup (seems to be getaddrinfo(3) via https://nodejs.org/api/dns.html#dnslookup). I don’t know exactly how that order is defined.

To test, I temporarily commented out ::1 localhost from /etc/hosts. Then Node.js 17 manages to make the request to my HTTP server only listening on ipv4 (0.0.0.0), as 127.0.0.1 is the only result returned. (Reordering the lines in /etc/hosts did not make a difference).

  • Node.js 17 no longer has an “astonishing” default, which seems good.
  • But people will accidentally have relied on the previous behavior, so there will be a transition period of fixing things.
  • One can use 127.0.0.1 to explicitly say “ipv4 localhost”.
  • Check the server you connect to: Does it serve on 0.0.0.0 (ipv4) or :: (ipv6) or both?
  • There’s a thing called Happy Eyeballs which means connecting to both ipv4 and ipv6 in parallel, which Node.js doesn’t have, but explains why other tools/languages can connect. Update 2023-01-27: Node.js now has Happy Eyeballs: net: add autoSelectFamily option #44731. Released in Node.js 20.

@treysis
Copy link
Contributor

treysis commented Apr 20, 2022

Yes. What is still surprising to me: why does Node sometimes seem to resolve localhost differently depending on if opening a listening socket and on opening a connect socket. At least on a local system it should be the same, no? I couldn't figure it out when introducing the PR last year that stopped reordering the results.

  • Yes, in general it's good.
  • That's why it wasn't backported to 16. You should expect changes with a new major version. But it was also introduced because more and more people started having problems that have to rely on IPv6. There were ambitions to introduce it much much earlier, probably as early as Node 12 (iirc), but it required a larger effort because the automated tests were already crafted with too much IPv4 in mind and because IPv6 connectivity of users back then wasn't always too reliable.
  • You could also specify family: 4. but 127.0.0.1 is well known and even IPv6-only systems will still use 127.0.0.1 for many more years, so it seems reasonable enough to just keep it for now for simplicity.
  • Unfortunately there's no default listening socket for dualstack on localhost. You can easily listen on :: and typically it should mean both 0.0.0.0 and ::, but you can't do that for localhost. You'd need to open a socket on 127.0.0.1 and a second one on ::1.
  • Unfortunately Happy Eyeballs is not as trivial as it sounds. But it's something that would also help in general DNS round robin situations.

devoto13 added a commit to devoto13/karma that referenced this issue May 10, 2022
Node 17+ changed the DNS resolution (see nodejs/node#40702), so now it resolves `localhost` according to the OS settings instead of IPv4-address first. The Karma server only listens on IPv4 address (127.0.0.1) by default, but the requests are sent to `localhost` in several places and `localhost` is resolved into IPv6 address (`::`) in Node 17+. So the run/stop/proxy request is unable to reach the Karma server and produces an error. This commit configures karma to use the IPv4-address first approach in newer Node version as well.

In the future major release, we may consider changing defaults to listen on IPv6 address instead, but IPv6 is not supported in Docker on macOS and Windows, so I think we should not rush such a change to make sure karma works there out of the box.

Fixes karma-runner#3730
devoto13 added a commit to devoto13/karma that referenced this issue May 10, 2022
Node 17+ changed the DNS resolution (see nodejs/node#40702), so now it resolves `localhost` according to the OS settings instead of IPv4-address first. The Karma server only listens on IPv4 address (127.0.0.1) by default, but the requests are sent to `localhost` in several places and `localhost` is resolved into IPv6 address (`::`) in Node 17+. So the run/stop/proxy request is unable to reach the Karma server and produces an error. This commit configures karma to use the IPv4-address first approach in newer Node version as well.

In the future major release, we may consider changing defaults to listen on IPv6 address instead, but IPv6 is not supported in Docker on macOS and Windows, so I think we should not rush such a change to make sure karma works there out of the box.

Fixes karma-runner#3730
@javierfuentesm
Copy link

@javierfuentesm Upgrade to node 20 where this has been resolved.

that worked! thank you!

@angelorc
Copy link

@javierfuentesm Upgrade to node 20 where this has been resolved.

Which version? I'm running on v20.5.1 and I still get the error

@treysis
Copy link
Contributor

treysis commented Dec 19, 2023

I still get the error

Which error exactly and what do you do?

@Rakeshbora007
Copy link

NEXT_PUBLIC_API_URL=http://127.0.0.1:3000

it still not working formw

@tahmidrahman-dsi
Copy link

Could anyone find this problem in Node v18.19.0 with Ubuntu 20.04.6?

@treysis
Copy link
Contributor

treysis commented Jan 19, 2024

NEXT_PUBLIC_API_URL=http://127.0.0.1:3000

it still not working formw

Which Node version? Did you try localhost instead of 127.0.0.1?

@treysis
Copy link
Contributor

treysis commented Jan 19, 2024

Could anyone find this problem in Node v18.19.0

Use Node 20!

@1-Rishav
Copy link

Make sure that your password and port number of database should be correct because this "ERROR" take place when your database credentials are wrong.
Screenshot 2024-01-27 105845

@mattdesl
Copy link

Is the code in the OP supposed to work on node 20?

const connect = (host, port) => {
  socket = new require('net').Socket();
  socket
    .connect(port, host)
    .on('connect', () => console.log('konected!'))
    .on('error', (e) => console.log(e));
};

connect('localhost', 5098);

I get the same error in node v20.11.1 and v21.6.2:

AggregateError [ECONNREFUSED]: 
    at internalConnectMultiple (node:net:1116:18)
    at afterConnectMultiple (node:net:1680:7) {
  code: 'ECONNREFUSED',
  [errors]: [
    Error: connect ECONNREFUSED 127.0.0.1:5098
        at createConnectionError (node:net:1643:14)
        at afterConnectMultiple (node:net:1673:16) {
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '127.0.0.1',
      port: 5098
    },
    Error: connect ECONNREFUSED ::1:5098
        at createConnectionError (node:net:1643:14)
        at afterConnectMultiple (node:net:1673:16) {
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '::1',
      port: 5098
    }
  ]
}

This also seems related to the error I'm getting from this demo app:
p2panda/mushroom-app-tutorial#7

@lydell
Copy link

lydell commented Feb 17, 2024

@mattdesl In your error message, you can see how there are two errors: Node tried both connecting via IPv4 127.0.0.1:5098, and via IPv6 ::1:5098, which is the Happy Eyeballs thing added in #44731.

I get that error too, because I have nothing running on port 5098. But when I start a server listening on 5098, OP’s code runs without errors:

require('http').createServer().listen(5098);

// If you do this instead, the below code fails on Node.js 16 but works on later Node.js versions (unsure exactly which releases):
// require('http').createServer().listen(5098, '::1');

const connect = (host, port) => {
  socket = new require('net').Socket();
  socket
    .connect(port, host)
    .on('connect', () => console.log('konected!'))
    .on('error', (e) => console.log(e));
};

connect('localhost', 5098);

@undergroundwires
Copy link

Here's my workaround (open-source and documented) that I hope that can help you too:

After days of research and trial/error, this is how I got this working:

  1. Create a script called force-ipv4.sh, that configures system to prefer IPv4 over IPv6, call it to configure the machine. It was not easy to find a reliable cross-platform solution and I went Cloudflare WARP for DNS resolution along with some system configurations.
  2. To easily use the script in GitHub workflows, create GitHub action called force-ipv4 and call it in GitHub runners.
  3. Fixes the IPv6 request issues, and you can happily run e.g. fetch API from Node.

Related commit introducing this fix: undergroundwires/privacy.sexy@52fadcd

undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 30, 2024
This commit upgrades Node.js version to v20.x in CI/CD environment.

Previously used Node 18.x is moving towards end-of-life, with a planned
date of 2025-04-30. In contrast, Node 20.x has been offering long-term
support (LTS) since 2023-10-24. This makes Node 20.x a stable and
recommended version for production environments.

This commit also configures `actions/setup-node` with the
`check-latest` flag to always use the latest Node 20.x version, keeping
CI/CD setup up-to-date with minimal maintenance.
Details:
- actions/setup-node#165
- actions/setup-node#160

Using Node 20.x in CI/CD environments provides better compatibility with
Electron v29.0 which moves to Node 20.x.
Details:
- electron/electron#40343

This upgrade improves network connection handling in CI/CD pipelines
(where issues occur due to GitHub runners not supporting IPv6).
Details:
- actions/runner#3138
- actions/runner-images#668
- actions/runner#3213
- actions/runner-images#9540

Node 20.x adopts the Happy Eyeballs algorithm for improved IPv6
connectivity.
- nodejs/node#40702
- nodejs/node#41625
- nodejs/node#44731

This mitigates issues like `UND_ERR_CONNECT_TIMEOUT` and localhost DNS
resolution in CI/CD environments:
Details:
- nodejs/node#40537
- actions/runner#3213
- actions/runner-images#9540

Node 20 introduces `setDefaultAutoSelectFamily`, a global function from
Node 19.4.0, enabling better IPv4 support, especially in environments
with limited or problematic IPv6 support.
Details:
- nodejs/node#45777

Node 20.x defaults to the new `autoSelectFamily`, improving network
connection reliability in GitHub runners lacking full IPv6 support.
Details:
- nodejs/node#46790
undergroundwires added a commit to undergroundwires/privacy.sexy that referenced this issue Mar 30, 2024
This commit upgrades Node.js version to v20.x in CI/CD environment.

Previously used Node 18.x is moving towards end-of-life, with a planned
date of 2025-04-30. In contrast, Node 20.x has been offering long-term
support (LTS) since 2023-10-24. This makes Node 20.x a stable and
recommended version for production environments.

This commit also configures `actions/setup-node` with the
`check-latest` flag to always use the latest Node 20.x version, keeping
CI/CD setup up-to-date with minimal maintenance.
Details:
- actions/setup-node#165
- actions/setup-node#160

Using Node 20.x in CI/CD environments provides better compatibility with
Electron v29.0 which moves to Node 20.x.
Details:
- electron/electron#40343

This upgrade improves network connection handling in CI/CD pipelines
(where issues occur due to GitHub runners not supporting IPv6).
Details:
- actions/runner#3138
- actions/runner-images#668
- actions/runner#3213
- actions/runner-images#9540

Node 20.x adopts the Happy Eyeballs algorithm for improved IPv6
connectivity.
- nodejs/node#40702
- nodejs/node#41625
- nodejs/node#44731

This mitigates issues like `UND_ERR_CONNECT_TIMEOUT` and localhost DNS
resolution in CI/CD environments:
Details:
- nodejs/node#40537
- actions/runner#3213
- actions/runner-images#9540

Node 20 introduces `setDefaultAutoSelectFamily`, a global function from
Node 19.4.0, enabling better IPv4 support, especially in environments
with limited or problematic IPv6 support.
Details:
- nodejs/node#45777

Node 20.x defaults to the new `autoSelectFamily`, improving network
connection reliability in GitHub runners lacking full IPv6 support.
Details:
- nodejs/node#46790
p2edwards added a commit to kobotoolbox/kpi that referenced this issue Jul 16, 2024
Here's the error we were encountering when running mocha-chrome
on Node >=17:

```
Promise Rejection:  Error: connect ECONNREFUSED ::1:39193
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1606:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '::1',
  port: 39193
}
```

Here's the patch that fixes this for us:

   // node_modules/mocha-chrome/lib/client.js
-  const client = await CDP({ port: instance.port });
+  const client = await CDP({ port: instance.port, host: '127.0.0.1' });

An alternative patch that would also succeed:

   // node_modules/chrome-remote-interface/lib/defaults.js
-  module.exports.HOST = 'localhost';
+  module.exports.HOST = '127.0.0.1';

Another option:

   // node_modules/chrome-remote-interface/lib/external-request.js
-  const {address} = await util.promisify(dns.lookup)(options.host);
+  const {address} = await util.promisify(dns.lookup)(
+    options.host,
+    {family:'IPv4'},
+  );

Thank you @nsainaney for writing this comment in nodejs/node#40702 [^1]:

> It appears to be a breaking change with how DNS.lookup works. With
> node 16, the lookup would return a IPv4 address but with node 17, it
> returns an IPv6 address which will break most REST clients that
> hardcode URLS like http://localhost:4040/api if the upstream server
> only binds to the IPv4 address (e.g. server.listen('127.0.0.1'…) etc…

[^1]: nodejs/node#40702 (comment)

Before committing to the 'patch' strategy, I checked to see if either
mocha-chrome or chrome-remote-interface had been updated later with this
workaround.

- chrome-remote-interface accepts a parameter for host, so nope.
  Makes sense, since you can specify the host manually.
- mocha-chrome doesn't parameterize host, and the author of mocha-chrome
  considers it an obsolete package and is no longer updating it.
journey-ad added a commit to journey-ad/Bitmagnet-Next-Web that referenced this issue Jul 23, 2024
journey-ad added a commit to journey-ad/Bitmagnet-Next-Web that referenced this issue Jul 23, 2024
journey-ad added a commit to journey-ad/Bitmagnet-Next-Web that referenced this issue Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
net Issues and PRs related to the net subsystem.
Projects
None yet
Development

No branches or pull requests