Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SocketError: other side closed #583

Closed
ShrimpAIO opened this issue Mar 2, 2021 · 48 comments · Fixed by #822
Closed

SocketError: other side closed #583

ShrimpAIO opened this issue Mar 2, 2021 · 48 comments · Fixed by #822
Labels
usage help [Use a Discussion instead]

Comments

@ShrimpAIO
Copy link

ShrimpAIO commented Mar 2, 2021

Upon connecting to a proxy with user:pass it's all good, but when I try to make a request to an proxy which is IP authenticated, I get 'SocketError: other side closed'.
I'm not sending the proxy-authorization header as it's not needed.
Blanked the proxy ip and port below.

var Client = require('undici');
(async ()=>{
	const client = new Client("http://proxyip:proxyport")
	const response = await client.request({
	  method: 'GET',
	  path: 'https://api.ipify.org?format=json',
	  headers: {
	    //'proxy-authorization': `Basic ${Buffer.from(':').toString('base64')}`
	  }
	})

	response.body.setEncoding('utf8')
	let data = ''
	for await (const chunk of response.body) {
	  data += chunk
	}
	//
	console.log(response.statusCode)
	console.log(JSON.parse(data))

	client.close()
})();
@Ethan-Arrowood
Copy link
Collaborator

@ShrimpAIO do you have an example with another HTTP client of this working?

@Ethan-Arrowood Ethan-Arrowood added the usage help [Use a Discussion instead] label Mar 2, 2021
@ShrimpAIO
Copy link
Author

@ShrimpAIO do you have an example with another HTTP client of this working?

var res = await superagent .get('https://api.ipify.org?format=json') .proxy(proxyUrl);
Blanked out the proxy above but the above code works on Superagent. Also worked on Axios & request-proxy. Just for some reason they refuse to work with Undici.

@ShrimpAIO
Copy link
Author

`node:internal/process/promises:245
triggerUncaughtException(err, true /* fromPromise */);
^

SocketError: other side closed
at Socket.onSocketEnd (D:\Libraries\Documents\Undici\node_modules\undici\lib\core\client.js:773:22)
at Socket.emit (node:events:390:22)
at endReadableNT (node:internal/streams/readable:1307:12)
at processTicksAndRejections (node:internal/process/task_queues:81:21) {
code: 'UND_ERR_SOCKET'
}`
Incase this is any help - here's a fuller error log

@Ethan-Arrowood
Copy link
Collaborator

So from what I can gather these other HTTP clients may be automatically adding things to the request for you. I'd recommend logging the complete request from something like superagent and see what else is being sent over. Compare that to undici and go from there.

@ShrimpAIO
Copy link
Author

ShrimpAIO commented Mar 2, 2021

So from what I can gather these other HTTP clients may be automatically adding things to the request for you. I'd recommend logging the complete request from something like superagent and see what else is being sent over. Compare that to undici and go from there.

I've made some progress but still not working with these particular proxies.
Adding browser-like headers to the request on Windows presents this:

<title>400 Bad Request</title>

400 Bad Request

400
Meanwhile, the same code on Ubuntu throws me this:

SocketError: other side closed
at Socket.onSocketEnd (/home/ubuntu/Crustacean/Monitors/Shopify/node_modules/undici/lib/core/client.js:773:22)
at Socket.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21) {
code: 'UND_ERR_SOCKET'
}

The IP's are 100% authenticated as the superagent code works both on Ubuntu and Windows..

For context, the code being tried now is as below;
const response = await client.request({
method: 'GET',
path: 'https://api.ipify.org/?format=json',
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0',
'Host': 'api.ipify.org',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8',
'Accept-Encoding': 'gzip, deflate, br',
'Origin': 'https://api.ipify.org'
}
})

@ShrimpAIO
Copy link
Author

Seems like another proxy provider refuses to work;
Bad Protocol
403
This one was user:pass authenticated.
There are two providers I have access to that fully work, but proxy support does seem iffy.

@Ethan-Arrowood
Copy link
Collaborator

Maybe this is something not yet achievable in Undici see #569

@Ethan-Arrowood
Copy link
Collaborator

And #568 might help you too

@ShrimpAIO
Copy link
Author

Ok;
The two issues above did not help, another discovery has however been made - only http requests can be successfully made, the moment a request is https these proxies do not like it and throw odd errors

@MysticEarth
Copy link

This also happened to me when making a normal request to for example "http://nvidia.com" using the example from the Quickstart. I do get some response but after that my app crashes:

response received 302
headers {
  location: 'http://www.nvidia.com/',
  'content-type': 'text/html',
  'cache-control': 'private',
  connection: 'close'
}

The application crashes on:

events.js:292
      throw er; // Unhandled 'error' event
      ^

SocketError: other side closed
    at Socket.onSocketEnd (/Users/<my-username>/<my-project>/node_modules/undici/lib/client.js:916:22)
    at Socket.emit (events.js:327:22)
    at endReadableNT (internal/streams/readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)
Emitted 'error' event on RequestResponse instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  code: 'UND_ERR_SOCKET'

@ronag
Copy link
Member

ronag commented Apr 12, 2021

Do you have the same problem with 4.0.0-alpha.2?

@MysticEarth
Copy link

Just tried and this still happend with 4.0.0-alpha.2

@ronag
Copy link
Member

ronag commented Apr 13, 2021

Can you please provide a repro example?

@MoritzLoewenstein
Copy link
Contributor

MoritzLoewenstein commented May 14, 2021

I had a pretty similar error:

node:events:342
      throw er; // Unhandled 'error' event
      ^

SocketError: other side closed
    at TLSSocket.onSocketEnd (/home/path/undici-fetch-repro/node_modules/undici/lib/core/client.js:781:22)
    at TLSSocket.emit (node:events:377:35)
    at endReadableNT (node:internal/streams/readable:1312:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on RequestResponse instance at:
    at emitErrorNT (node:internal/streams/destroy:193:8)
    at emitErrorCloseNT (node:internal/streams/destroy:158:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'UND_ERR_SOCKET'
}

I build a small reproduction, its just requesting a lot of files one by one from cloudfront.
See here: https://github.com/MoritzLoewenstein/undici-fetch-repro

Note: this does happen consistently but it might take a few requests.

Edit: Nevermind, when I use undici & not undici-fetch in my reproduction example, this error doesnt happen anymore.

Edit2:
So, I investigated a bit more, it has something to do with the images on my cloudfront / s3. When I upload a random image (or .txt file), it doesnt happen, but the images already on there produce the error. I already tried changing the "Content-Type" of the image from "application/octet-stream" to "image/png", but it still produces the error.
You can try it with this image: d1h7ilwuof4uhg.cloudfront.net
Sometimes it happens after 2 tries, sometimes after 500 ¯\(ツ)
Maybe a stupid idea, but is it possible that "content-length" is not accurate and causing a failure?

@MoritzLoewenstein
Copy link
Contributor

grafik
(pinging because you asked for a reproduction) @ronag
Can anyone confirm this issue with my provided reproduction repo?
If its helpful i can create a new issue but I think its the same underlying problem, and should be labeled a bug.

@mcollina
Copy link
Member

@MoritzLoewenstein Could you use a Node.js server to reproduce?

@MoritzLoewenstein
Copy link
Contributor

Hey, I added a node server. The issue is indeed an invalid Content-Length header, if I don't set the Content-Length header on the server I cant reproduce the issue.
When I copy the Content-Length value from the cloudfront / s3 response it will crash after 10-15k tries.

@mcollina
Copy link
Member

The issue is indeed an invalid Content-Length header

Do you know what's the value of this?

@MoritzLoewenstein
Copy link
Contributor

MoritzLoewenstein commented May 21, 2021

When I dont set the Content-Length header, node will set the Transfer-Encoding header to chunked (= no Content-Length header).
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
22661 is apparently the correct filesize for the test image:
grafik
Edit: to be clear i just found this out, i just assumed node would set the correct Content-Length header before.

@mcollina
Copy link
Member

Can you prepare an example to reproduce the problem? You can also use a net server and send out HTTP in text form.

@MoritzLoewenstein
Copy link
Contributor

MoritzLoewenstein commented May 21, 2021

My provided example should be sufficient to show the problem: https://github.com/MoritzLoewenstein/undici-fetch-repro
It doesnt reproduce the problem on every request, but after around 10k to 15k requests the uncaughtException occurs.

I investigated a bit more: When the code enters onSocketEnd in client.js, the error will happen 100% of the time (but onSocketEnd doesnt happen everytime).

here is the parser printed in this function:
<ref *1> Parser {
ptr: 88736,
client: <ref *2> Client {
 _events: [Object: null prototype] {
   drain: [Function: onDrain],
   connect: [Function (anonymous)],
   disconnect: [Function (anonymous)]
 },
 _eventsCount: 3,
 _maxListeners: undefined,
 [Symbol(kCapture)]: false,
 [Symbol(url)]: URL {
   href: 'http://localhost:3000/',
   origin: 'http://localhost:3000',
   protocol: 'http:',
   username: '',
   password: '',
   host: 'localhost:3000',
   hostname: 'localhost',
   port: '3000',
   pathname: '/',
   search: '',
   searchParams: URLSearchParams {},
   hash: ''
 },
 [Symbol(connector)]: [Function: bound connect],
 [Symbol(socket)]: Socket {
   connecting: false,
   _hadError: false,
   _parent: null,
   _host: 'localhost',
   _readableState: [ReadableState],
   _events: [Object: null prototype],
   _eventsCount: 4,
   _maxListeners: undefined,
   _writableState: [WritableState],
   allowHalfOpen: false,
   _sockname: null,
   _pendingData: null,
   _pendingEncoding: '',
   server: null,
   _server: null,
   write: [Function: writeAfterFIN],
   [Symbol(async_id_symbol)]: 11,
   [Symbol(kHandle)]: [TCP],
   [Symbol(kSetNoDelay)]: true,
   [Symbol(lastWriteQueueSize)]: 0,
   [Symbol(timeout)]: null,
   [Symbol(kBuffer)]: null,
   [Symbol(kBufferCb)]: null,
   [Symbol(kBufferGen)]: null,
   [Symbol(kCapture)]: false,
   [Symbol(kBytesRead)]: 0,
   [Symbol(kBytesWritten)]: 0,
   [Symbol(no ref)]: false,
   [Symbol(connecting)]: false,
   [Symbol(writing)]: false,
   [Symbol(reset)]: false,
   [Symbol(error)]: null,
   [Symbol(parser)]: [Circular *1],
   [Symbol(client)]: [Circular *2]
 },
 [Symbol(pipelinig)]: 1,
 [Symbol(max headers size)]: 16384,
 [Symbol(connect timeout value)]: 10000,
 [Symbol(default keep alive timeout)]: 4000,
 [Symbol(max keep alive timeout)]: 600000,
 [Symbol(keep alive timeout threshold)]: 1000,
 [Symbol(keep alive timeout)]: 4000,
 [Symbol(closed)]: false,
 [Symbol(destroyed)]: false,
 [Symbol(server name)]: null,
 [Symbol(destroy callbacks)]: [],
 [Symbol(resuming)]: 0,
 [Symbol(need drain)]: 2,
 [Symbol(host header)]: 'host: localhost:3000\r\n',
 [Symbol(body timeout)]: 30000,
 [Symbol(headers timeout)]: 30000,
 [Symbol(strict content length)]: true,
 [Symbol(queue)]: [ [Request] ],
 [Symbol(running index)]: 0,
 [Symbol(pending index)]: 1,
 [Symbol(needDrain)]: true
},
socket: <ref *3> Socket {
 connecting: false,
 _hadError: false,
 _parent: null,
 _host: 'localhost',
 _readableState: ReadableState {
   objectMode: false,
   highWaterMark: 16384,
   buffer: BufferList { head: null, tail: null, length: 0 },
   length: 0,
   pipes: [],
   flowing: false,
   ended: true,
   endEmitted: true,
   reading: false,
   constructed: true,
   sync: false,
   needReadable: false,
   emittedReadable: false,
   readableListening: false,
   resumeScheduled: false,
   errorEmitted: false,
   emitClose: false,
   autoDestroy: true,
   destroyed: false,
   errored: null,
   closed: false,
   closeEmitted: false,
   defaultEncoding: 'utf8',
   awaitDrainWriters: null,
   multiAwaitDrain: false,
   readingMore: false,
   decoder: null,
   encoding: null,
   [Symbol(kPaused)]: true
 },
 _events: [Object: null prototype] {
   end: [Array],
   error: [Function: onSocketError],
   data: [Function: onSocketData],
   close: [Function: onSocketClose]
 },
 _eventsCount: 4,
 _maxListeners: undefined,
 _writableState: WritableState {
   objectMode: false,
   highWaterMark: 16384,
   finalCalled: false,
   needDrain: false,
   ending: false,
   ended: false,
   finished: false,
   destroyed: false,
   decodeStrings: false,
   defaultEncoding: 'utf8',
   length: 0,
   writing: false,
   corked: 0,
   sync: false,
   bufferProcessing: false,
   onwrite: [Function: bound onwrite],
   writecb: null,
   writelen: 0,
   afterWriteTickInfo: null,
   buffered: [],
   bufferedIndex: 0,
   allBuffers: true,
   allNoop: true,
   pendingcb: 0,
   constructed: true,
   prefinished: false,
   errorEmitted: false,
   emitClose: false,
   autoDestroy: true,
   errored: null,
   closed: false,
   closeEmitted: false,
   [Symbol(kOnFinished)]: []
 },
 allowHalfOpen: false,
 _sockname: null,
 _pendingData: null,
 _pendingEncoding: '',
 server: null,
 _server: null,
 write: [Function: writeAfterFIN],
 [Symbol(async_id_symbol)]: 11,
 [Symbol(kHandle)]: TCP {
   reading: true,
   onconnection: null,
   [Symbol(owner_symbol)]: [Circular *3]
 },
 [Symbol(kSetNoDelay)]: true,
 [Symbol(lastWriteQueueSize)]: 0,
 [Symbol(timeout)]: null,
 [Symbol(kBuffer)]: null,
 [Symbol(kBufferCb)]: null,
 [Symbol(kBufferGen)]: null,
 [Symbol(kCapture)]: false,
 [Symbol(kBytesRead)]: 0,
 [Symbol(kBytesWritten)]: 0,
 [Symbol(no ref)]: false,
 [Symbol(connecting)]: false,
 [Symbol(writing)]: false,
 [Symbol(reset)]: false,
 [Symbol(error)]: null,
 [Symbol(parser)]: [Circular *1],
 [Symbol(client)]: <ref *2> Client {
   _events: [Object: null prototype],
   _eventsCount: 3,
   _maxListeners: undefined,
   [Symbol(kCapture)]: false,
   [Symbol(url)]: [URL],
   [Symbol(connector)]: [Function: bound connect],
   [Symbol(socket)]: [Circular *3],
   [Symbol(pipelinig)]: 1,
   [Symbol(max headers size)]: 16384,
   [Symbol(connect timeout value)]: 10000,
   [Symbol(default keep alive timeout)]: 4000,
   [Symbol(max keep alive timeout)]: 600000,
   [Symbol(keep alive timeout threshold)]: 1000,
   [Symbol(keep alive timeout)]: 4000,
   [Symbol(closed)]: false,
   [Symbol(destroyed)]: false,
   [Symbol(server name)]: null,
   [Symbol(destroy callbacks)]: [],
   [Symbol(resuming)]: 0,
   [Symbol(need drain)]: 2,
   [Symbol(host header)]: 'host: localhost:3000\r\n',
   [Symbol(body timeout)]: 30000,
   [Symbol(headers timeout)]: 30000,
   [Symbol(strict content length)]: true,
   [Symbol(queue)]: [Array],
   [Symbol(running index)]: 0,
   [Symbol(pending index)]: 1,
   [Symbol(needDrain)]: true
 }
},
timeout: Timeout {
 _idleTimeout: 30000,
 _idlePrev: [Timeout],
 _idleNext: [TimersList],
 _idleStart: 61,
 _onTimeout: [Function: onParserTimeout],
 _timerArgs: [Array],
 _repeat: null,
 _destroyed: false,
 [Symbol(refed)]: false,
 [Symbol(kHasPrimitive)]: false,
 [Symbol(asyncId)]: 18,
 [Symbol(triggerId)]: 16
},
timeoutValue: 30000,
timeoutType: 2,
statusCode: 200,
upgrade: false,
headers: [],
headersSize: 0,
headersMaxSize: 16384,
shouldKeepAlive: true,
paused: true,
resume: [Function: bound resume],
bytesRead: 22661,
trailer: '',
keepAlive: 'timeout=5',
contentLength: '22661'
}

@mcollina
Copy link
Member

It's not enough for a unit test.

@MoritzLoewenstein
Copy link
Contributor

Alright, I will see if I can narrow it down.

@mcollina
Copy link
Member

Alright, I will see if I can narrow it down.

Thanks! Tracking down those conditions is hard :(.

@MoritzLoewenstein
Copy link
Contributor

The reason undici crashes is an incorrect Transfer-Encoding header on the server:
per default my test server would not set this header at all, but apparently still send the content
according to Transfer-Encoding: "chunked".
If I manually set the header on the server, everything works fine.

One more thing I found out in the process, undici will keep the node process open if
Connection: "keep-alive" & Keep-Alive: "timeout=5" headers are set (for as long as the timeout lasts).
node-fetch will close instantly after the request has finished (not sure who is "wrong" or if this is even a problem).

Current reproduction with server & files is here: https://github.com/MoritzLoewenstein/undici-fetch-repro

@MoritzLoewenstein
Copy link
Contributor

MoritzLoewenstein commented Jun 5, 2021

Ok, I have written a test that will fail, but I think I cant push it because of that.

here is the code for the test:
test('dont crash on unexpected Transfer-Encoding Header', async (t) => {
  t.plan(2)

 const server = createServer((req, res) => {
   res.setHeader('Content-Type', 'text/plain')

   res.removeHeader('Connection')
   res.removeHeader('Keep-Alive')
   res.removeHeader('Transfer-Encoding')
   const str = 'Chunked Transfer Encoding Test'
   res.write(str.repeat(832))
   res.end(str)
 })
 t.teardown(server.close.bind(server))

 server.listen(0, async () => {
   const client = new Client(`http://localhost:${server.address().port}`)
   t.teardown(client.close.bind(client))

   process.on('uncaughtException', () => {
     t.fail('uncaughtException happened')
   })
   try {
     await client.request({
       path: '/',
       method: 'GET'
     })
     t.pass('request successful')
   } catch (err) {
     t.pass('error happened')
   } finally {
     process.removeListener('uncaughtException')
   }
 })
})

mcollina added a commit that referenced this issue Jun 6, 2021
@mcollina
Copy link
Member

mcollina commented Jun 6, 2021

Your test does not run successfully, it has some issues with promises, callbacks and how tap needs things laid out.

However, I have adapted your test to run and it passes on master (see #822).
Have you tried v4.0.0-rc.5?

My understanding is that this is fixed.

@mcollina
Copy link
Member

mcollina commented Jun 6, 2021

(You can push a failing test with git commit -n)

@MoritzLoewenstein
Copy link
Contributor

MoritzLoewenstein commented Jun 6, 2021

Im sorry for the incorrect test, but the issue still happens on rc5. Apparently the issue only occurs when not doing anything with the response body, because you called body.resume at the end and it doesnt happen in that scenario. Also just reading and printing the body prevents it:

const { body } = await undici.request("http://localhost:3000");
  for await (const data of body) {
    console.log("data", data);
  }

I dont know how common this is but my original use case was to just read the response headers, status code etc and doing nothing with the body.

@ronag
Copy link
Member

ronag commented Jun 6, 2021

You need to consume or destroy the response body. I think the docs mention this.

@MoritzLoewenstein
Copy link
Contributor

I wasnt aware of that and couldnt find it in the docs, but if that is the case it should be handled in undici-fetch i guess?

@ronag
Copy link
Member

ronag commented Jun 6, 2021

Yea I can’t find it either now. Maybe got lost when we did the docs overhaul. PR welcome! All response bodies must always be fully consumed or destroyed.

@istarkov
Copy link

istarkov commented Jun 11, 2021

I'm still having this bug with 4.0.0-rc.7, what is wrong?

import type { RequestHandler } from '@sveltejs/kit';
import pkg from 'undici';
const { Pool } = pkg;

// process.env.API_ORIGIN would be killed by vite (bug), so hack
const client = new Pool(
  process['env']['API_ORIGIN'] ?? 'http://localhost:6543',
);

export const post: RequestHandler = async req => {
  // We are waiting this https://github.com/sveltejs/kit/issues/1563
  // to complete, then we must switch to previous implementation, preserving compression etc
  try {
    const headers: Record<string, string | string[]> = {
      'content-type': 'application/json; charset=utf-8',
    };

    const data = await client.request({
      path: '/v1/graphql',
      method: 'POST',
      body: JSON.stringify(req.body),
      headers,
    });

    let strBody = '';
    for await (const chunk of data.body) {
      strBody += chunk;
    }

    const allowedHeaders = [
      'content-encoding',
      'content-type',
    ];

    const headersOut: Record<string, string> = {};

    for (const allowedheader of allowedHeaders) {
      const headerValue = data.headers[allowedheader] as string;
      if (headerValue != null) {
        headersOut[allowedheader] = headerValue;
      }
    }

    return {
      headers: headersOut,
      body: strBody,
    };
  } catch (e) {
    console.error('ERROR: error occured at graphql.js handler', e.message);
    console.error(e);
    throw e;
  }
};

PS:
No issues with node-fetch@next,
no issues when 'Accept-Encoding': 'gzip, deflate' is used

@moises-marquez
Copy link

I'm facing a similar error, but with fastify-http-proxy which uses undici underneath. The server stops with an unrecoverable error when the request is suddenly closed:

{"level":40,"time":1629317990531,"pid":12743,"hostname":"moises-minna","reqId":"req-q","name":"SocketError","code":"UND_ERR_SOCKET","stack":"SocketError: other side closed\n    at Socket.onSocketEnd (/my_project/node_modules/undici/lib/client.js:1050:22)\n    at Socket.emit (events.js:388:22)\n    at Socket.emit (domain.js:470:12)\n    at endReadableNT (internal/streams/readable.js:1336:12)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21)","type":"Error","msg":"response errored"}

/my_project/node_modules/fastify/lib/request.js:150
      return this.socket.remoteAddress
                         ^
TypeError: Cannot read property 'remoteAddress' of null
    at _Request.get (/my_project/node_modules/fastify/lib/request.js:150:26)
    at Object.asReqValue [as req] (/my_project/node_modules/fastify/lib/logger.js:55:26)
    at Pino.asJson (/my_project/node_modules/pino/lib/tools.js:118:50)
    at Pino.write (/my_project/node_modules/pino/lib/proto.js:201:28)
    at Pino.LOG [as error] (/my_project/node_modules/pino/lib/tools.js:55:21)
    at defaultErrorHandler (/my_project/node_modules/fastify/fastify.js:81:15)
    at handleError (/my_project/node_modules/fastify/lib/reply.js:550:20)
    at onErrorHook (/my_project/node_modules/fastify/lib/reply.js:521:5)
    at _Reply.Reply.send (/my_project/node_modules/fastify/lib/reply.js:127:5)
    at onErrorDefault (/my_project/node_modules/fastify-reply-from/index.js:209:9)

@titanism
Copy link
Contributor

titanism commented Jun 5, 2023

Hi there, still experiencing this issue on undici v5.22.1. Any resolution here?

@titanism
Copy link
Contributor

titanism commented Jun 5, 2023

@ronag @mcollina Here's my findings so far (this occurs when you run a for loop with repeated undici.request attempts to GET some data from a locally running server). There's nothing magic going on here, just using undici to crawl a sitemap and using a for loop with await undici.request(...) inside of it (using default options and only supplying the path from sitemap).

I modified the culprit of this error being thrown here:

function onSocketEnd () {
  const { [kParser]: parser } = this

  if (parser.statusCode && !parser.shouldKeepAlive) {
    // We treat all incoming data so far as a valid response.
    parser.onMessageComplete()
    return
  }

+  console.log('parser', parser);
+  console.log('socket info', util.getSocketInfo(this));
+  console.log('this', this);

+  // this causes the error to be thrown
  util.destroy(this, new SocketError('other side closed', util.getSocketInfo(this)))
}

The output of parser:

parser <ref *1> Parser {
  llhttp: [Object: null prototype] {
    memory: Memory [WebAssembly.Memory] {},
    _initialize: [Function: 9],
    __indirect_function_table: Table [WebAssembly.Table] {},
    llhttp_init: [Function: 10],
    llhttp_should_keep_alive: [Function: 65],
    llhttp_alloc: [Function: 12],
    malloc: [Function: 70],
    llhttp_free: [Function: 13],
    free: [Function: 72],
    llhttp_get_type: [Function: 14],
    llhttp_get_http_major: [Function: 15],
    llhttp_get_http_minor: [Function: 16],
    llhttp_get_method: [Function: 17],
    llhttp_get_status_code: [Function: 18],
    llhttp_get_upgrade: [Function: 19],
    llhttp_reset: [Function: 20],
    llhttp_execute: [Function: 21],
    llhttp_settings_init: [Function: 22],
    llhttp_finish: [Function: 23],
    llhttp_pause: [Function: 24],
    llhttp_resume: [Function: 25],
    llhttp_resume_after_upgrade: [Function: 26],
    llhttp_get_errno: [Function: 27],
    llhttp_get_error_reason: [Function: 28],
    llhttp_set_error_reason: [Function: 29],
    llhttp_get_error_pos: [Function: 30],
    llhttp_errno_name: [Function: 31],
    llhttp_method_name: [Function: 32],
    llhttp_status_name: [Function: 33],
    llhttp_set_lenient_headers: [Function: 34],
    llhttp_set_lenient_chunked_length: [Function: 35],
    llhttp_set_lenient_keep_alive: [Function: 36],
    llhttp_set_lenient_transfer_encoding: [Function: 37],
    llhttp_message_needs_eof: [Function: 63]
  },
  ptr: 141936,
  client: <ref *2> Client {
    _events: [Object: null prototype] {
      drain: [Function: onDrain],
      connect: [Function (anonymous)],
      disconnect: [Function (anonymous)],
      connectionError: [Function (anonymous)]
    },
    _eventsCount: 4,
    _maxListeners: undefined,
    [Symbol(kCapture)]: false,
    [Symbol(destroyed)]: false,
    [Symbol(onDestroyed)]: null,
    [Symbol(closed)]: false,
    [Symbol(onClosed)]: [],
    [Symbol(dispatch interceptors)]: [ [Function (anonymous)] ],
    [Symbol(url)]: URL {
      href: 'http://localhost:3000/',
      origin: 'http://localhost:3000',
      protocol: 'http:',
      username: '',
      password: '',
      host: 'localhost:3000',
      hostname: 'localhost',
      port: '3000',
      pathname: '/',
      search: '',
      searchParams: URLSearchParams {},
      hash: ''
    },
    [Symbol(connector)]: [Function: connect],
    [Symbol(socket)]: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'localhost',
      _closeAfterHandlingError: false,
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      write: [Function: writeAfterFIN],
      [Symbol(async_id_symbol)]: 144,
      [Symbol(kHandle)]: [TCP],
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: true,
      [Symbol(kSetKeepAlive)]: true,
      [Symbol(kSetKeepAliveInitialDelay)]: 60,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(no ref)]: false,
      [Symbol(writing)]: false,
      [Symbol(reset)]: false,
      [Symbol(blocking)]: false,
      [Symbol(error)]: null,
      [Symbol(parser)]: [Circular *1],
      [Symbol(client)]: [Circular *2],
      [Symbol(socket request counter)]: 0,
      [Symbol(maxRequestsPerClient)]: undefined
    },
    [Symbol(pipelining)]: 1,
    [Symbol(max headers size)]: 16384,
    [Symbol(default keep alive timeout)]: 4000,
    [Symbol(max keep alive timeout)]: 600000,
    [Symbol(keep alive timeout threshold)]: 1000,
    [Symbol(keep alive timeout)]: 4000,
    [Symbol(server name)]: null,
    [Symbol(local address)]: null,
    [Symbol(resuming)]: 0,
    [Symbol(need drain)]: 2,
    [Symbol(host header)]: 'host: localhost:3000\r\n',
    [Symbol(body timeout)]: 300000,
    [Symbol(headers timeout)]: 300000,
    [Symbol(strict content length)]: true,
    [Symbol(maxRedirections)]: undefined,
    [Symbol(maxRequestsPerClient)]: undefined,
    [Symbol(kClosedResolve)]: null,
    [Symbol(max response size)]: -1,
    [Symbol(queue)]: [ [Request] ],
    [Symbol(running index)]: 0,
    [Symbol(pending index)]: 1,
    [Symbol(Intercepted Dispatch)]: [Function: Intercept],
    [Symbol(connecting)]: false,
    [Symbol(needDrain)]: true
  },
  socket: <ref *3> Socket {
    connecting: false,
    _hadError: false,
    _parent: null,
    _host: 'localhost',
    _closeAfterHandlingError: false,
    _readableState: ReadableState {
      objectMode: false,
      highWaterMark: 65536,
      buffer: BufferList { head: null, tail: null, length: 0 },
      length: 0,
      pipes: [],
      flowing: false,
      ended: true,
      endEmitted: true,
      reading: false,
      constructed: true,
      sync: false,
      needReadable: false,
      emittedReadable: false,
      readableListening: true,
      resumeScheduled: false,
      errorEmitted: false,
      emitClose: false,
      autoDestroy: true,
      destroyed: false,
      errored: null,
      closed: false,
      closeEmitted: false,
      defaultEncoding: 'utf8',
      awaitDrainWriters: null,
      multiAwaitDrain: false,
      readingMore: false,
      dataEmitted: true,
      decoder: null,
      encoding: null,
      [Symbol(kPaused)]: null
    },
    _events: [Object: null prototype] {
      end: [Array],
      error: [Array],
      readable: [Function: onSocketReadable],
      close: [Function: onSocketClose]
    },
    _eventsCount: 4,
    _maxListeners: undefined,
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 65536,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: false,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: false,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: false,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    allowHalfOpen: false,
    _sockname: null,
    _pendingData: null,
    _pendingEncoding: '',
    server: null,
    _server: null,
    write: [Function: writeAfterFIN],
    [Symbol(async_id_symbol)]: 144,
    [Symbol(kHandle)]: TCP {
      reading: true,
      onconnection: null,
      [Symbol(owner_symbol)]: [Circular *3]
    },
    [Symbol(lastWriteQueueSize)]: 0,
    [Symbol(timeout)]: null,
    [Symbol(kBuffer)]: null,
    [Symbol(kBufferCb)]: null,
    [Symbol(kBufferGen)]: null,
    [Symbol(kCapture)]: false,
    [Symbol(kSetNoDelay)]: true,
    [Symbol(kSetKeepAlive)]: true,
    [Symbol(kSetKeepAliveInitialDelay)]: 60,
    [Symbol(kBytesRead)]: 0,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(no ref)]: false,
    [Symbol(writing)]: false,
    [Symbol(reset)]: false,
    [Symbol(blocking)]: false,
    [Symbol(error)]: null,
    [Symbol(parser)]: [Circular *1],
    [Symbol(client)]: <ref *2> Client {
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      [Symbol(kCapture)]: false,
      [Symbol(destroyed)]: false,
      [Symbol(onDestroyed)]: null,
      [Symbol(closed)]: false,
      [Symbol(onClosed)]: [],
      [Symbol(dispatch interceptors)]: [Array],
      [Symbol(url)]: [URL],
      [Symbol(connector)]: [Function: connect],
      [Symbol(socket)]: [Circular *3],
      [Symbol(pipelining)]: 1,
      [Symbol(max headers size)]: 16384,
      [Symbol(default keep alive timeout)]: 4000,
      [Symbol(max keep alive timeout)]: 600000,
      [Symbol(keep alive timeout threshold)]: 1000,
      [Symbol(keep alive timeout)]: 4000,
      [Symbol(server name)]: null,
      [Symbol(local address)]: null,
      [Symbol(resuming)]: 0,
      [Symbol(need drain)]: 2,
      [Symbol(host header)]: 'host: localhost:3000\r\n',
      [Symbol(body timeout)]: 300000,
      [Symbol(headers timeout)]: 300000,
      [Symbol(strict content length)]: true,
      [Symbol(maxRedirections)]: undefined,
      [Symbol(maxRequestsPerClient)]: undefined,
      [Symbol(kClosedResolve)]: null,
      [Symbol(max response size)]: -1,
      [Symbol(queue)]: [Array],
      [Symbol(running index)]: 0,
      [Symbol(pending index)]: 1,
      [Symbol(Intercepted Dispatch)]: [Function: Intercept],
      [Symbol(connecting)]: false,
      [Symbol(needDrain)]: true
    },
    [Symbol(socket request counter)]: 0,
    [Symbol(maxRequestsPerClient)]: undefined
  },
  timeout: Timeout {
    callback: [Function: onParserTimeout],
    delay: 300000,
    opaque: [Circular *1],
    state: 1685997171039
  },
  timeoutValue: 300000,
  timeoutType: 2,
  statusCode: 200,
  statusText: 'OK',
  upgrade: false,
  headers: [],
  headersSize: 0,
  headersMaxSize: 16384,
  shouldKeepAlive: true,
  paused: true,
  resume: [Function: bound resume],
  bytesRead: 73102,
  keepAlive: 'timeout=5',
  contentLength: '73102',
  connection: 'keep-alive',
  maxResponseSize: -1
}

The output of socket info:

socket info {
  localAddress: '127.0.0.1',
  localPort: 63749,
  remoteAddress: '127.0.0.1',
  remotePort: 3000,
  remoteFamily: 'IPv4',
  timeout: undefined,
  bytesWritten: 66,
  bytesRead: 76703
}

The output of this:

this <ref *1> Socket {
  connecting: false,
  _hadError: false,
  _parent: null,
  _host: 'localhost',
  _closeAfterHandlingError: false,
  _readableState: ReadableState {
    objectMode: false,
    highWaterMark: 65536,
    buffer: BufferList { head: null, tail: null, length: 0 },
    length: 0,
    pipes: [],
    flowing: false,
    ended: true,
    endEmitted: true,
    reading: false,
    constructed: true,
    sync: false,
    needReadable: false,
    emittedReadable: false,
    readableListening: true,
    resumeScheduled: false,
    errorEmitted: false,
    emitClose: false,
    autoDestroy: true,
    destroyed: false,
    errored: null,
    closed: false,
    closeEmitted: false,
    defaultEncoding: 'utf8',
    awaitDrainWriters: null,
    multiAwaitDrain: false,
    readingMore: false,
    dataEmitted: true,
    decoder: null,
    encoding: null,
    [Symbol(kPaused)]: null
  },
  _events: [Object: null prototype] {
    end: [ [Function: onReadableStreamEnd], [Function: onSocketEnd] ],
    error: [ [Function (anonymous)], [Function: onSocketError] ],
    readable: [Function: onSocketReadable],
    close: [Function: onSocketClose]
  },
  _eventsCount: 4,
  _maxListeners: undefined,
  _writableState: WritableState {
    objectMode: false,
    highWaterMark: 65536,
    finalCalled: false,
    needDrain: false,
    ending: false,
    ended: false,
    finished: false,
    destroyed: false,
    decodeStrings: false,
    defaultEncoding: 'utf8',
    length: 0,
    writing: false,
    corked: 0,
    sync: false,
    bufferProcessing: false,
    onwrite: [Function: bound onwrite],
    writecb: null,
    writelen: 0,
    afterWriteTickInfo: null,
    buffered: [],
    bufferedIndex: 0,
    allBuffers: true,
    allNoop: true,
    pendingcb: 0,
    constructed: true,
    prefinished: false,
    errorEmitted: false,
    emitClose: false,
    autoDestroy: true,
    errored: null,
    closed: false,
    closeEmitted: false,
    [Symbol(kOnFinished)]: []
  },
  allowHalfOpen: false,
  _sockname: { address: '127.0.0.1', family: 'IPv4', port: 63749 },
  _pendingData: null,
  _pendingEncoding: '',
  server: null,
  _server: null,
  write: [Function: writeAfterFIN],
  _peername: { address: '127.0.0.1', family: 'IPv4', port: 3000 },
  [Symbol(async_id_symbol)]: 144,
  [Symbol(kHandle)]: TCP {
    reading: true,
    onconnection: null,
    [Symbol(owner_symbol)]: [Circular *1]
  },
  [Symbol(lastWriteQueueSize)]: 0,
  [Symbol(timeout)]: null,
  [Symbol(kBuffer)]: null,
  [Symbol(kBufferCb)]: null,
  [Symbol(kBufferGen)]: null,
  [Symbol(kCapture)]: false,
  [Symbol(kSetNoDelay)]: true,
  [Symbol(kSetKeepAlive)]: true,
  [Symbol(kSetKeepAliveInitialDelay)]: 60,
  [Symbol(kBytesRead)]: 0,
  [Symbol(kBytesWritten)]: 0,
  [Symbol(no ref)]: false,
  [Symbol(writing)]: false,
  [Symbol(reset)]: false,
  [Symbol(blocking)]: false,
  [Symbol(error)]: null,
  [Symbol(parser)]: <ref *2> Parser {
    llhttp: [Object: null prototype] {
      memory: Memory [WebAssembly.Memory] {},
      _initialize: [Function: 9],
      __indirect_function_table: Table [WebAssembly.Table] {},
      llhttp_init: [Function: 10],
      llhttp_should_keep_alive: [Function: 65],
      llhttp_alloc: [Function: 12],
      malloc: [Function: 70],
      llhttp_free: [Function: 13],
      free: [Function: 72],
      llhttp_get_type: [Function: 14],
      llhttp_get_http_major: [Function: 15],
      llhttp_get_http_minor: [Function: 16],
      llhttp_get_method: [Function: 17],
      llhttp_get_status_code: [Function: 18],
      llhttp_get_upgrade: [Function: 19],
      llhttp_reset: [Function: 20],
      llhttp_execute: [Function: 21],
      llhttp_settings_init: [Function: 22],
      llhttp_finish: [Function: 23],
      llhttp_pause: [Function: 24],
      llhttp_resume: [Function: 25],
      llhttp_resume_after_upgrade: [Function: 26],
      llhttp_get_errno: [Function: 27],
      llhttp_get_error_reason: [Function: 28],
      llhttp_set_error_reason: [Function: 29],
      llhttp_get_error_pos: [Function: 30],
      llhttp_errno_name: [Function: 31],
      llhttp_method_name: [Function: 32],
      llhttp_status_name: [Function: 33],
      llhttp_set_lenient_headers: [Function: 34],
      llhttp_set_lenient_chunked_length: [Function: 35],
      llhttp_set_lenient_keep_alive: [Function: 36],
      llhttp_set_lenient_transfer_encoding: [Function: 37],
      llhttp_message_needs_eof: [Function: 63]
    },
    ptr: 141936,
    client: Client {
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      [Symbol(kCapture)]: false,
      [Symbol(destroyed)]: false,
      [Symbol(onDestroyed)]: null,
      [Symbol(closed)]: false,
      [Symbol(onClosed)]: [],
      [Symbol(dispatch interceptors)]: [Array],
      [Symbol(url)]: [URL],
      [Symbol(connector)]: [Function: connect],
      [Symbol(socket)]: [Circular *1],
      [Symbol(pipelining)]: 1,
      [Symbol(max headers size)]: 16384,
      [Symbol(default keep alive timeout)]: 4000,
      [Symbol(max keep alive timeout)]: 600000,
      [Symbol(keep alive timeout threshold)]: 1000,
      [Symbol(keep alive timeout)]: 4000,
      [Symbol(server name)]: null,
      [Symbol(local address)]: null,
      [Symbol(resuming)]: 0,
      [Symbol(need drain)]: 2,
      [Symbol(host header)]: 'host: localhost:3000\r\n',
      [Symbol(body timeout)]: 300000,
      [Symbol(headers timeout)]: 300000,
      [Symbol(strict content length)]: true,
      [Symbol(maxRedirections)]: undefined,
      [Symbol(maxRequestsPerClient)]: undefined,
      [Symbol(kClosedResolve)]: null,
      [Symbol(max response size)]: -1,
      [Symbol(queue)]: [Array],
      [Symbol(running index)]: 0,
      [Symbol(pending index)]: 1,
      [Symbol(Intercepted Dispatch)]: [Function: Intercept],
      [Symbol(connecting)]: false,
      [Symbol(needDrain)]: true
    },
    socket: [Circular *1],
    timeout: Timeout {
      callback: [Function: onParserTimeout],
      delay: 300000,
      opaque: [Circular *2],
      state: 1685997171039
    },
    timeoutValue: 300000,
    timeoutType: 2,
    statusCode: 200,
    statusText: 'OK',
    upgrade: false,
    headers: [],
    headersSize: 0,
    headersMaxSize: 16384,
    shouldKeepAlive: true,
    paused: true,
    resume: [Function: bound resume],
    bytesRead: 73102,
    keepAlive: 'timeout=5',
    contentLength: '73102',
    connection: 'keep-alive',
    maxResponseSize: -1
  },
  [Symbol(client)]: Client {
    _events: [Object: null prototype] {
      drain: [Function: onDrain],
      connect: [Function (anonymous)],
      disconnect: [Function (anonymous)],
      connectionError: [Function (anonymous)]
    },
    _eventsCount: 4,
    _maxListeners: undefined,
    [Symbol(kCapture)]: false,
    [Symbol(destroyed)]: false,
    [Symbol(onDestroyed)]: null,
    [Symbol(closed)]: false,
    [Symbol(onClosed)]: [],
    [Symbol(dispatch interceptors)]: [ [Function (anonymous)] ],
    [Symbol(url)]: URL {
      href: 'http://localhost:3000/',
      origin: 'http://localhost:3000',
      protocol: 'http:',
      username: '',
      password: '',
      host: 'localhost:3000',
      hostname: 'localhost',
      port: '3000',
      pathname: '/',
      search: '',
      searchParams: URLSearchParams {},
      hash: ''
    },
    [Symbol(connector)]: [Function: connect],
    [Symbol(socket)]: [Circular *1],
    [Symbol(pipelining)]: 1,
    [Symbol(max headers size)]: 16384,
    [Symbol(default keep alive timeout)]: 4000,
    [Symbol(max keep alive timeout)]: 600000,
    [Symbol(keep alive timeout threshold)]: 1000,
    [Symbol(keep alive timeout)]: 4000,
    [Symbol(server name)]: null,
    [Symbol(local address)]: null,
    [Symbol(resuming)]: 0,
    [Symbol(need drain)]: 2,
    [Symbol(host header)]: 'host: localhost:3000\r\n',
    [Symbol(body timeout)]: 300000,
    [Symbol(headers timeout)]: 300000,
    [Symbol(strict content length)]: true,
    [Symbol(maxRedirections)]: undefined,
    [Symbol(maxRequestsPerClient)]: undefined,
    [Symbol(kClosedResolve)]: null,
    [Symbol(max response size)]: -1,
    [Symbol(queue)]: [ [Request] ],
    [Symbol(running index)]: 0,
    [Symbol(pending index)]: 1,
    [Symbol(Intercepted Dispatch)]: [Function: Intercept],
    [Symbol(connecting)]: false,
    [Symbol(needDrain)]: true
  },
  [Symbol(socket request counter)]: 0,
  [Symbol(maxRequestsPerClient)]: undefined
}

Hope this helps you debug this issue further.

@titanism
Copy link
Contributor

titanism commented Jun 5, 2023

It looks like it's throwing due to parser.shouldKeepAlive being true due to the keep-alive header having a value in our server's response.

@titanism
Copy link
Contributor

titanism commented Jun 5, 2023

I've tried setting options keepAlive, keepalive, pipelining to values of 0, null, and false, and nothing seems to disable this behavior.

const { statusCode, headers } = await request(url.loc, {
  signal: AbortSignal.timeout(10000),
  throwOnError: true,
  pipelining: 0
});

@titanism
Copy link
Contributor

titanism commented Jun 5, 2023

Shouldn't this be changed?

undici/lib/client.js

Lines 989 to 993 in a3e0fdc

if (parser.statusCode && !parser.shouldKeepAlive) {
// We treat all incoming data so far as a valid response.
parser.onMessageComplete()
return
}

To:

-  if (parser.statusCode && !parser.shouldKeepAlive) {
+  if (parser.statusCode || !parser.shouldKeepAlive) {
    // We treat all incoming data so far as a valid response.
    parser.onMessageComplete()
    return
  }

@titanism
Copy link
Contributor

titanism commented Jun 5, 2023

Additional findings:

  • It seems like after the 8th request, it just freezes.
  • No combination of pipelining: 0 or other approaches above works (even when initialized via const client = new Client('...', { pipelining: 0 });
  • Server is responding just fine to requests outside of undici (e.g. via curl)
  • As soon as you set { pipelining: 0 } the requests just hang and do not complete any further past the 8th request. If you remove { pipelining: 0 } option then the error SocketError: other side closed is throw
  • Calling body.destroy() after a request is made causes the loop to throw and exit early with AbortError: Request aborted if you have { signal: AbortSignal.timeout(10000) } for example. This is probably a bug - if the user invokes body.destroy() then the signal should not be considered to be aborted? Very confusing anti-patterns going on here.

@titanism
Copy link
Contributor

titanism commented Jun 5, 2023

The fix is to call await body.text() after every response.... 🤦

This definitely seems like a major anti-pattern @mcollina @ronag. We don't explicitly state anywhere in the undici docs to call a body mixin (otherwise this error will occur). I think there is two things that need done:

  1. There is most likely a bug here SocketError: other side closed #583 (comment). Also see our note about body.destroy() not working in combination with signal.
  2. Invoking a body mixin method such as body.text() should not be required in order to keep requests flowing.

@GopherJ
Copy link

GopherJ commented Nov 8, 2023

I dont think this issue should be closed. @mcollina

@metcoder95
Copy link
Member

This issue has been closed since 3 years ago; please open a new one with a description and an
Minimum Reproducible Example if possible 🙂

@WilfredAlmeida
Copy link

I'm still facing the issue in 2024.

For anyone pulling their hair over this, this issue is not at all encountered with the bun runtime, give it a shot.

@iscekic
Copy link

iscekic commented Jul 10, 2024

Any update on this? I'm consuming the body with await body.json(), yet this error still seems to be intermittently reported.

@WilfredAlmeida
Copy link

I have managed to fix this and other socket errors in my applications by limiting the nuber of connections NodeJS is allowed to open to the server. I observed that the node process was opening too many connections to the server, that is, using too many local ports of the host (16000+ in my case) and eventually these connections would end abruptly and cause the error.

The number of connections can be limited via setting the connections param in the httpAgent

https.Agent({
  maxSockets: 50,
})

50 might not be an ideal number for you and you might still face the issue or your application might slow down because the process has lesser number of connections. You need to trial and error and find a number that works for you. 100 worked well for me when I did over 2 million async API calls for testing the limits.

You can monitor the number of ports used by your node process on unix based systems via the following command

lsof -i -n -P | grep node | awk '{print $9}' | awk -F '->' '{print $1}' | awk -F ':' '{print $2}' | sort -u | wc -l

Bun used around 400-500 ports and was faster than NodeJS. I've tested using node v20.13.1 and bun 1.1.8 on my local

I haven't dug into whether the OS limits the number of connections, runs out of them or NodeJS is unable to handle them

If this doesn't fix the issue for you and you want to dig in further, a packet capture can be helpful. I did a pcap and analyzed the packets in Wireshark. You'd want to look for the flow, source, destination, origin of the SYN, SYN/ACK, RST packets. The expert analysis tool in Wireshark does a great job at adding clarity as well.

Also, if the server you're connecting to has rate limits, give a look at what it does to new connection requests and exising connections once the limits are hit

Overall, this issue is now fixed for me and for all of the people I've recommended the above mentioned fix.

@aleluff
Copy link

aleluff commented Sep 28, 2024

The bug can popup when the disk is full also

@WilfredAlmeida
Copy link

@aleluff This may be one of the cause as well. I had plenty of disk space available during working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
usage help [Use a Discussion instead]
Projects
None yet
Development

Successfully merging a pull request may close this issue.