Clients read config from file resulting in EMFILE (too many open files) errors

[moltar]

Describe the bug

This applies to every auto-generated client, but I will use client-sqs as an example.

This is related to: #2271, #2027, #2993

---

Note that in the above snippet, loadNodeConfig is called multiple times to get the defaults for the client configuration:

aws-sdk-js-v3/clients/client-sqs/src/runtimeConfig.ts

Lines 31 to 53 in bda3b40

	return {
	...clientSharedValues,
	...config,
	runtime: "node",
	base64Decoder: config?.base64Decoder ?? fromBase64,
	base64Encoder: config?.base64Encoder ?? toBase64,
	bodyLengthChecker: config?.bodyLengthChecker ?? calculateBodyLength,
	credentialDefaultProvider:
	config?.credentialDefaultProvider ?? decorateDefaultCredentialProvider(credentialDefaultProvider),
	defaultUserAgentProvider:
	config?.defaultUserAgentProvider ??
	defaultUserAgent({ serviceId: clientSharedValues.serviceId, clientVersion: packageInfo.version }),
	maxAttempts: config?.maxAttempts ?? loadNodeConfig(NODE_MAX_ATTEMPT_CONFIG_OPTIONS),
	md5: config?.md5 ?? Hash.bind(null, "md5"),
	region: config?.region ?? loadNodeConfig(NODE_REGION_CONFIG_OPTIONS, NODE_REGION_CONFIG_FILE_OPTIONS),
	requestHandler: config?.requestHandler ?? new NodeHttpHandler(),
	retryMode: config?.retryMode ?? loadNodeConfig(NODE_RETRY_MODE_CONFIG_OPTIONS),
	sha256: config?.sha256 ?? Hash.bind(null, "sha256"),
	streamCollector: config?.streamCollector ?? streamCollector,
	useDualstackEndpoint: config?.useDualstackEndpoint ?? loadNodeConfig(NODE_USE_DUALSTACK_ENDPOINT_CONFIG_OPTIONS),
	useFipsEndpoint: config?.useFipsEndpoint ?? loadNodeConfig(NODE_USE_FIPS_ENDPOINT_CONFIG_OPTIONS),
	utf8Decoder: config?.utf8Decoder ?? fromUtf8,
	utf8Encoder: config?.utf8Encoder ?? toUtf8,

Which is the exported fn loadConfig from @aws-sdk/node-config-provider package:

aws-sdk-js-v3/packages/node-config-provider/src/configLoader.ts

Lines 27 to 37 in 7392341

	export const loadConfig = <T = string>(
	{ environmentVariableSelector, configFileSelector, default: defaultValue }: LoadedConfigSelectors<T>,
	configuration: LocalConfigOptions = {}
	): Provider<T> =>
	memoize(
	chain(
	fromEnv(environmentVariableSelector),
	fromSharedConfigFiles(configFileSelector, configuration),
	fromStatic(defaultValue)
	)
	);

Which then uses fromSharedConfigFiles to load config values from the disk.

This is a very undesirable "feature", especially in the serverless environments.

Because under heavy load this results in the following errors:

    error: {
      "type": "NodeError",
      "message": "A system error occurred: uv_os_homedir returned EMFILE (too many open files)",
      "stack":
          SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_homedir returned EMFILE (too many open files)
              at Object.getHomeDir (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:82:17)
              at Object.loadSharedConfigFiles (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:11:89)
              at null.<anonymous> (/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53)
              at null.<anonymous> (/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28)
              at runMicrotasks (<anonymous>)
              at processTicksAndRejections (internal/process/task_queues.js:95:5)
              at null.coalesceProvider (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
              at Object.isConstant (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)
              at Object.getEndpointFromRegion (/node_modules/@aws-sdk/config-resolver/dist-cjs/endpointsConfig/utils/getEndpointFromRegion.js:12:34)
              at null.buildHttpRpcRequest (/node_modules/@aws-sdk/client-sqs/dist-cjs/protocols/Aws_query.js:2540:68)
      "code": "ERR_SYSTEM_ERROR",
      "info": {
        "errno": -24,
        "code": "EMFILE",
        "message": "too many open files",
        "syscall": "uv_os_homedir"
      },
      "errno": -24,
      "syscall": "uv_os_homedir"
    }

---

Yes the call is memoized, but if your Lambda is getting executed heavily, and installation happens within the handler, then this call happens multiple times.

Your environment

SDK version number

@aws-sdk/client-sqs@3.40.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

node -v v14.17.5

Steps to reproduce

import { SQS } from '@aws-sdk/client-sqs'

const sqs = new SQS({})

Observed behavior

    error: {
      "type": "NodeError",
      "message": "A system error occurred: uv_os_homedir returned EMFILE (too many open files)",
      "stack":
          SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_homedir returned EMFILE (too many open files)
              at Object.getHomeDir (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:82:17)
              at Object.loadSharedConfigFiles (/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:11:89)
              at null.<anonymous> (/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53)
              at null.<anonymous> (/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28)
              at runMicrotasks (<anonymous>)
              at processTicksAndRejections (internal/process/task_queues.js:95:5)
              at null.coalesceProvider (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
              at Object.isConstant (/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)
              at Object.getEndpointFromRegion (/node_modules/@aws-sdk/config-resolver/dist-cjs/endpointsConfig/utils/getEndpointFromRegion.js:12:34)
              at null.buildHttpRpcRequest (/node_modules/@aws-sdk/client-sqs/dist-cjs/protocols/Aws_query.js:2540:68)
      "code": "ERR_SYSTEM_ERROR",
      "info": {
        "errno": -24,
        "code": "EMFILE",
        "message": "too many open files",
        "syscall": "uv_os_homedir"
      },
      "errno": -24,
      "syscall": "uv_os_homedir"
    }

Expected behavior

1. Do not error out
2. Do not read config from disk by default, or allow overriding this behaviour.

Screenshots

N/A

Additional context

N/A

[moltar]

Added a console.log to slurpFile:

aws-sdk-js-v3/packages/shared-ini-file-loader/src/index.ts

Line 100 in bda3b40

const slurpFile = (path: string): Promise<string> =>

const slurpFile = (path) => new Promise((resolve, reject) => {
    console.log(path, new Error().stack)

And tested with the following two-liner file:

import { SQS } from '@aws-sdk/client-sqs'
const sqs = new SQS({})

Results

/Users/user/.aws/config Error: 
    at /Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:66:23
    at new Promise (<anonymous>)
    at slurpFile (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:64:29)
    at Object.loadSharedConfigFiles (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:13:9)
    at defaultProvider (/Users/user/project/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js:17:57)
    at Object.credentialDefaultProvider (/Users/user/project/node_modules/@aws-sdk/client-sts/dist-cjs/defaultRoleAssumers.js:10:68)
    at Object.resolveAwsAuthConfig (/Users/user/project/node_modules/@aws-sdk/middleware-signing/dist-cjs/configurations.js:10:17)
    at new SQSClient (/Users/user/project/node_modules/@aws-sdk/client-sqs/dist-cjs/SQSClient.js:20:48)
    at new SQS (/Users/user/project/node_modules/@aws-sdk/client-sqs/dist-cjs/SQS.js:25:1)
    at Object.<anonymous> (/Users/user/project/test/sdk.ts:3:13)
/Users/user/.aws/credentials Error: 
    at /Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:66:23
    at new Promise (<anonymous>)
    at slurpFile (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:64:29)
    at Object.loadSharedConfigFiles (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:14:9)
    at defaultProvider (/Users/user/project/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js:17:57)
    at Object.credentialDefaultProvider (/Users/user/project/node_modules/@aws-sdk/client-sts/dist-cjs/defaultRoleAssumers.js:10:68)
    at Object.resolveAwsAuthConfig (/Users/user/project/node_modules/@aws-sdk/middleware-signing/dist-cjs/configurations.js:10:17)
    at new SQSClient (/Users/user/project/node_modules/@aws-sdk/client-sqs/dist-cjs/SQSClient.js:20:48)
    at new SQS (/Users/user/project/node_modules/@aws-sdk/client-sqs/dist-cjs/SQS.js:25:1)
    at Object.<anonymous> (/Users/user/project/test/sdk.ts:3:13)
/Users/user/.aws/config Error: 
    at /Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:66:23
    at new Promise (<anonymous>)
    at slurpFile (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:64:29)
    at Object.loadSharedConfigFiles (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:13:9)
    at /Users/user/project/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53
    at /Users/user/project/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at coalesceProvider (/Users/user/project/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
    at isConstant (/Users/user/project/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)
/Users/user/.aws/credentials Error: 
    at /Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:66:23
    at new Promise (<anonymous>)
    at slurpFile (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:64:29)
    at Object.loadSharedConfigFiles (/Users/user/project/node_modules/@aws-sdk/shared-ini-file-loader/dist-cjs/index.js:14:9)
    at /Users/user/project/node_modules/@aws-sdk/node-config-provider/dist-cjs/fromSharedConfigFiles.js:9:53
    at /Users/user/project/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at coalesceProvider (/Users/user/project/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
    at isConstant (/Users/user/project/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:24:28)

As you can see, it is trying to read the same files twice, and makes at least 4 IO operations per instance of a client.

[moltar]

Now, making this a bit more realistic and instantiating SQS client in a loop.

This would be the case when it is instantiated inside a lambda handler.

Well, it wouldn't be a loop, but it would be instantiated multiple times when lambda is being executed warm, and very frequently.

import { SQS } from '@aws-sdk/client-sqs'

for (const iter of [1, 2]) {
  const sqs = new SQS({})
}

Then:

ts-node test/sdk.ts | grep Error | wc -l

Results

In 8 executions!

So the results are never cached globally, and are simply cached per client instance.

[moltar]

Here's a way to remove unnecessary calls:

import { SQS } from '@aws-sdk/client-sqs'

const sqs = new SQS({
  // removes first 2 calls
  credentials: {
    accessKeyId: 'x',
    secretAccessKey: '',
  },

  // removes second 2 calls
  defaultUserAgentProvider: async () => [],
})

[moltar]

But wait ... there is more!

If you actually try to use the methods of the queue, this will result in even more calls!

import { SQS } from '@aws-sdk/client-sqs'

const sqs = new SQS({
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID || '',
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY || '',
    sessionToken: process.env.AWS_SESSION_TOKEN || '',
  },
  region: 'us-east-1',
  defaultUserAgentProvider: async () => [],
})

async function main() {
  await sqs.listQueues({})
}

main()

Results

$> ts-node test/sdk.ts | grep Error | wc -l

8

So we get 8 IO calls for each method call (I tried sendMessage too, it's the same)!

This is ridiculous and needs to be addressed immediately, as it is causing major issues!

[moltar]

Interestingly, calling listQueues in a loop, does not cause additional calls. So, I guess the results do get cached.

[moltar]

If anyone stumbles on this, the only way I found to shut the file reading, is to do the following hack.

import * as sharedIniFileLoader from '@aws-sdk/shared-ini-file-loader'

Object.assign(sharedIniFileLoader, {
  loadSharedConfigFiles: async (): Promise<sharedIniFileLoader.SharedConfigFiles> => ({
    configFile: {},
    credentialsFile: {},
  }),
})

[petermorlion]

I can confirm that @moltar 's workaround solves this issue in AWS Lambda functions.

[adrai]

The aws-sdk should detect if running in lambda and omit the loadSharedConfigFiles call completely, or at least provide a "supported" way to disable the loadSharedConfigFiles call...

[adrai]

Without this workaround suggested by @moltar it seems v3.49.0 is even getting worse, and opening more file descriptors: probably because of this #3192 ?
//cc @AllanZhengYP @trivikr

[adrai]

Ok, it seems with v3.49.0 it seems these type of EMFILE errors, can still occur (also with that workaround):

Error: connect EMFILE 52.94.5.100:443 - Local (undefined:undefined)
Error: getaddrinfo EMFILE dynamodb.eu-west-1.amazonaws.com
Error: A system error occurred: uv_os_homedir returned EMFILE (too many open files)

PS. so having a lambda setup, like this is incrementing the probability for such EMFILE errors: https://github.com/fastify/aws-lambda-fastify#lower-cold-start-latency