Default AWS credentials provider chain resolution changed in v3 for EC2MetadataCredentials/ECSCredentials - readFile calls more frequent #2027
Description
Describe the bug
After upgrading an ECS(Fargate) from aws-sdk-js-v2 to v3, I started seeing a lot more calls to readFile being performed (in newrelic, per transaction) despite no environment variables changing.
The usage is effectively const { DynamoDB } = require('@aws-sdk/client-dynamodb'); /*... */; const db = new DynamoDB({region: 'us-east-1'}); , and the instance of DynamoDB is reused
I suspect this is because the existence of the AWS_PROFILE environment variable changes the credentials provider chain that is used - This was not the case in v2.
(I still haven't confirmed it)
- EDIT: It seems like I was mislead. AWS_PROFILE is only used when building the application in question for ECS, not when running the containers on ECS. Maybe it's some other issue such as credentials expiration not being set or creating too many brand new connections, but I can't imagine what
- EDIT: Before and after installing v3, the application calls aws-sdk v2's
new AWS.SecretsManager({endpoint, region})before other calls to aws but that doesn't seem relevant EDIT: I wonder if v3 is parsing the wrong expiry somehow - e.g. aws-sdk v2's node_modules/aws-sdk/lib/credentials/remote_credentials.js hadexpireTime: new Date(credData.expiration || credData.Expiration). I also see the ini files checked before remote credentials- EDIT: Strangely, it seems as if the calls to dynamodb are successful, or at least I haven't seen errors
- EDIT(2021-03-14): Credentials from EKS service account #1808 (comment) mentions why - "The TokenFileWebIdentityCredentials provider was added in v2 in aws/aws-sdk-js 2737 ; It's missing in JS SDK v3, as called out in our blog post." (It wasn't obvious to me which provider the v2 sdk was using in Fargate)
I don't see any documentation of why the decision was made to have ENV_PROFILE('AWS_PROFILE') be used this way in the file itself. It seems to have been done this way in ecb884b or earlier and kept that way (EDIT: Probably not the cause of this issue - AWS_PROFILE is not included in https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html and the aws console does not show it among additional environment variables)
- E.g. adding a line comment would help
- I didn't see this change to the credentials provider chain mentioned when glancing the migration notes for v3, but may have just missed it.
V3 credential chain is conditional
// aws-sdk-js-v3/packages/credential-provider-node/src/index.ts
// Observed: The providerChain used unexpectedly depends on AWS_PROFILE.
// If the environment variable AWS_PROFILE is set, then the automatic ECS credentials provider is no longer in the credentials chain?
export function defaultProvider(init: FromIniInit & RemoteProviderInit & FromProcessInit = {}): CredentialProvider {
const { profile = process.env[ENV_PROFILE] } = init;
const providerChain = profile
? chain(fromIni(init), fromProcess(init))
: chain(fromEnv(), fromIni(init), fromProcess(init), remoteProvider(init));
return memoize(
providerChain,
(credentials) => credentials.expiration !== undefined && credentials.expiration.getTime() - Date.now() < 300000,
(credentials) => credentials.expiration !== undefined
);
}V2 Credential Chain has no conditions
// node_modules/aws-sdk/lib/node_loader.js
// Setup default chain providers
// If this changes, please update documentation for
// AWS.CredentialProviderChain.defaultProviders in
// credentials/credential_provider_chain.js
AWS.CredentialProviderChain.defaultProviders = [
function () { return new AWS.EnvironmentCredentials('AWS'); },
function () { return new AWS.EnvironmentCredentials('AMAZON'); },
function () { return new AWS.SharedIniFileCredentials(); },
function () { return new AWS.ECSCredentials(); },
function () { return new AWS.ProcessCredentials(); },
function () { return new AWS.TokenFileWebIdentityCredentials(); },
function () { return new AWS.EC2MetadataCredentials(); }
];Your environment
SDK version number
@aws-sdk/client-dynamodb@3.4.1 (3.4.1 for everything else)
Is the issue in the browser/Node.js/ReactNative?
Nodejs
Details of the browser/Node.js/ReactNative version
node -v is roughly v14.15.5 (using Docker node:14)
Steps to reproduce
TODO
Observed behavior
Overall performance is worse, requests take longer to process after upgrading from v2 to v3. Calls to Filesystem.readFile are very frequent, probably trying to look up aws credentials in a missing file or failing to cache them.
Expected behavior
No performance impact.
If the environment variable AWS_PROFILE is set, this should not prevent the ECS/EC2 metadata service from being used if the required environment variables are available (i.e. chain(fromIni(init), fromProcess(init)) currently does not contain fromRemoteProvider)
Screenshots
(A spike in calls to readFile(green) was seen when upgrading to aws-sdk-js-v3 and went away after rolling back)
Additional context
Add any other context about the problem here.