Leaking file descriptors when generating too many http requests

[adrai]

This issue is extracted from this original issue: #3019, like requested: #3019 (comment)

#3019 is concentrated to the readFile (leaking file descriptor) "issue" and this issue is concentrated to the network (leaking file descriptor) "issue"

---

It seems lambda is not "waiting" for the file descriptors to be closed.

This can be observed especially, when having warm lambda executions with a lot of sdk calls, like for DynamoDb or S3, etc...
This each http request opens a network socket which results in an open file descriptor.
Since by default in Node.js the socket timeout is set to 120000ms (2 minutes) it may be the lambda is already finished, but the sockets are still open. When "restarting" the lambda for the next invocations, those file descriptors may still be open.
This leads to this type of EMFILE errors:

Error: connect EMFILE 52.94.5.100:443 - Local (undefined:undefined)
Error: getaddrinfo EMFILE dynamodb.eu-west-1.amazonaws.com
Error: A system error occurred: uv_os_homedir returned EMFILE (too many open files)

These basic tests shows the count (and leaks) of the emfile count:

Tests originally done in this issue here: #3019 (comment)

Details

compared to this tests: #3019 (comment)

Defining a custom requestHandler, with a very low socketTimeout reduces drastically the emfiles count:

  requestHandler: new NodeHttpHandler({
      socketTimeout: 10000 // <- this decreases the emfiles count, the Node.js default is 120000
  })

Details

That's why I suggest to set a low socket Timeout by default, like proposed here: #3019 (comment)

proposal:

// https://github.com/aws/aws-sdk-js-v3/blob/main/packages/node-http-handler/src/node-http-handler.ts#L62
socketTimeout: socketTimeout || 10000

and probably also here?

// https://github.com/aws/aws-sdk-js-v3/blob/main/packages/node-http-handler/src/node-http2-handler.ts#L44
this.requestTimeout = requestTimeout || 10000;

PS. btw. it seems it got worse (more file descriptors) when updating from v3.46.0 to v3.49.0

[adrai]

maybe also relevant: #2569 @artur-ma ?

[trivikr]

The fix in #3285 is most likely going to fix this issue too, as the number of file reads is reduced from 16 to 2 for client+operation call.

ToDo: Reproduce this issue using repo https://github.com/samswen/lambda-emfiles when the fix from #3285 is out.

[adrai]

The fix in #3285 is most likely going to fix this issue too, as the number of file reads is reduced from 16 to 2 for client+operation call.

ToDo: Reproduce this issue using repo https://github.com/samswen/lambda-emfiles when the fix from #3285 is out.

No, I don't think so.
This is not about readFile, but about http requests.
That's why you told me to open a new issue.

[adrai]

This issue (http requests) is also more important for you @ffxsam, right?

[trivikr]

No, I don't think so.

Yup, I'm also not sure and this request will require a deep dive.
That's why I did not mark this issue as fixes, but refs in the PR. And mentioned that we'll try to repro with https://github.com/samswen/lambda-emfiles once fix is out.

This is not about readFile, but about http requests.

Looking at the bug description, it appears to be because of readFile because of the following reasons:

• EMFILE errors are thrown when there are too many open files.
• It seems it got worse (more file descriptors) when updating from v3.46.0 to v3.49.0, where we added another call to readFile call for defaults.

That's why you told me to open a new issue.

The reason I asked to create a separate request is for better tracking.

[adrai]

fyi: the tests I did where without extra readFile calls, so these are all coming from http requests.
So the tests run with:

import sharedIniFileLoader from '@aws-sdk/shared-ini-file-loader'
Object.assign(sharedIniFileLoader, { loadSharedConfigFiles: async () => ({ configFile: {}, credentialsFile: {} }) })

That's why my proposal is to have a very low socketTimeout by default.
In my tests setting a low socketTimeout decreased the EMFILE count from up to over 600 to under 100 => 6x.

[trivikr]

PS. btw. it seems it got worse (more file descriptors) when updating from v3.46.0 to v3.49.0
In my tests setting a low socketTimeout decreased the EMFILE count from up to over 600 to under 100 => 6x.

This clearly shows that most of the EMFILE errors are caused by file reads being fixed in #3285

• Updating from v3.46.0 to v3.49.0 added additional readFile call to read defaults.
• The test setting removes readFile calls for all configurations (except credentials).

The default socketTimeout for the SDK will be application to all environments.
I'll revisit this issue after #3285 is out (most likely will be released with v3.51.0 on 02/11)

[adrai]

v3.51.0 has been released, any update regarding #3279 / #3291 ?