Error while using aws-secretsmanager with aws-java-sdk-sts

[jainhitesh9998]

Hi,
I'm working to deploy spring boot applicaiton with aws-secretsmanager-jdbc integrated. It works fine with the the usual flow where I set AWS_ACCESS_KEY_ID and and VALUE in env variable. when i'm trying to deploy the app in eks, ive created a service role with access to a particular secret, and included aws-java-sdk-sts to assume the role which is injected in the pod.
when i'm not adding the sdk-sts to pom, it tries to assume the role of the managed nodes, which doesn't have access to secrets. whenever I add the above library, it is able to assume the proper role but gives the below stack trace.

Caused by: java.lang.StackOverflowError: null
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(Unknown Source)
        at java.base/java.security.SecureClassLoader.defineClass(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1369)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1338)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1327)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:488)
        at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:460)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73) ```

is there a way to use AWS_WEB_IDENTITY_TOKEN_FILE instead of aws creds to access the secrets

[ktzevelekidis]

Hello,
I am facing a similar issue. In my case, even when I add the sts module on the classpath (in my application's build.gradle), it still does not use the correct credentials provider (i.e. WebIdentityTokenFileCredentialsProvider). It uses the node's IAM role.

Did you manage to find a solution?

I was thinking to download the library, add the sts dependency in the pom.xml, build it and then use (hoping that something will change).

[jainhitesh9998]

I've checked this library, it uses a pretty old version of aws as of now which doesn't support web identity token so even including sts module won't belp in this case. This needs an update to use compatible aws sdk version. As of now I've not found a solution for this. I'll be mostly using at sealed secrets for the time being. May also look into modifications needed in this code to support sts

[ktzevelekidis]

I added this dependency in my build.gradle:

• com.amazonaws:aws-java-sdk-sts:1.11.1004

and it worked for me. However, I get an exception the first time the application starts:

com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

As a consequence the application restarts and then it works.

[jainhitesh9998]

So it's using the node's iam role instead of the service account injected into pod right. I didn't want the node to have dB access so decided to dropped it. Was running short of time so had to look into alternative approaches

[ktzevelekidis]

No, it is using the service account role and not the node's role. The only issue is that in the first boot, I get the aforementioned exception, and a restart takes place. After this it works as expected. I will investigate further and I will inform you for my findings. Anyway, it would be good if the library will be migrated to aws sdk v2. I saw that there is an open PR for the migration from November, but not yet reviewed.

[jainhitesh9998]

@ktzevelekidis you mentioned initially that it was picking nodes iam role can you clarify more. For me when I add sts module which supports it, the pod just throws a stack trace. It looked to me like circular dependency error but it never managed to work

[ktzevelekidis]

Maybe I confused you. So, let me explain.

If the sts dependency is not present in the classpath (i.e. not inside build.gradle for my case), it will use the node's IAM role, which is something we do not want, as you also said. When we add the sts dependency, it uses the service account role. In order to achieve this with this specific library, it seems that you need to add the v1 of this sts dependency (i.e. com.amazonaws:aws-java-sdk-sts:1.11.1004).
I am using version 1.0.6 of the aws-secretsmanager-jdbc library

I hope it is more clear now!

[willtong1234]

Thank you for opening this issue - we are looking into it.

[KarakiBusiness]

I'm getting the same error - when I use the DefaultAWSCredentialsProviderChain locally, and I provide only the assumed role via the AWS_ROLE_ARN and AWS_ROLE_SESSION_NAME , I get the error, but when I also give it an access key and secret, it works as expected. I'm testing a deploy to ECS now and getting this error.

[galvezlj]

still not working for service role?

[itlicious]

We have the same issue on EKS. AWS_ROLE_ARN is set through a service-account. But it still uses the assumed role of the node.

[spirosag]

@ktzevelekidis

Maybe I confused you. So, let me explain.

If the sts dependency is not present in the classpath (i.e. not inside build.gradle for my case), it will use the node's IAM role, which is something we do not want, as you also said. When we add the sts dependency, it uses the service account role. In order to achieve this with this specific library, it seems that you need to add the v1 of this sts dependency (i.e. com.amazonaws:aws-java-sdk-sts:1.11.1004). I am using version 1.0.6 of the aws-secretsmanager-jdbc library

I hope it is more clear now!

This needs to be made clear and more visible. Thanks for the solution.

[simonmarty]

For those of you facing issues, could you tell me

• whether your cluster has IAM Roles for Service Accounts set up. The instructions to do that are here
• whether the AWS Java SDK in your dependency closure (mvn dependency:tree) is newer than 1.11.704, in order to support IRSA

We're going to document this to make it clearer how to ensure you are using the correct role for calls to Secrets Manager in this library. The behavior described in the issue seems consistent with other AWS APIs and libraries. Please let us know if you find discrepancies that suggest otherwise.

[simonkarman]

Hey @simonmarty,

I'm running into this issue too. Meaning: When I do NOT include sts dependecy, it tries to use the IAM role of the node. When I do include the 2.x version of the sts dependecy I get the following error message:

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@3eec8583: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@52657d5f: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]

You mentioned

The behavior described in the issue seems consistent with other AWS APIs and libraries.

However, I do see inconsistent behaviour. As, with the sts library v2 I was able to use the software.amazon.awssdk:* libraries correctly using the EKS service account role and I have used the following imports in gradle for that:

    implementation 'software.amazon.awssdk:secretsmanager:2.21.0'
    implementation 'software.amazon.awssdk:sts:2.21.0'
    implementation 'software.amazon.awssdk:s3:2.21.0'
    implementation 'software.amazon.awssdk:dynamodb:2.21.0'

However, the 'com.amazonaws.secretsmanager:aws-secretsmanager-jdbc:1.0.6' library seems to unfortunately be an exception to this.

[simonkarman]

Hey @simonmarty,

I'm running into this issue too. Meaning: When I do NOT include sts dependecy, it tries to use the IAM role of the node. When I do include the 2.x version of the sts dependecy I get the following error message:

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@3eec8583: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@52657d5f: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)]

You mentioned

The behavior described in the issue seems consistent with other AWS APIs and libraries.

However, I do see inconsistent behaviour. As, with the sts library v2 I was able to use the software.amazon.awssdk:* libraries correctly using the EKS service account role and I have used the following imports in gradle for that:

    implementation 'software.amazon.awssdk:secretsmanager:2.21.0'
    implementation 'software.amazon.awssdk:sts:2.21.0'
    implementation 'software.amazon.awssdk:s3:2.21.0'
    implementation 'software.amazon.awssdk:dynamodb:2.21.0'

However, the 'com.amazonaws.secretsmanager:aws-secretsmanager-jdbc:1.0.6' library seems to unfortunately be an exception to this.

Apparently I mixed to different version of the sts library. There is software.amazon.awssdk:sts and com.amazonaws:aws-java-sdk-sts. Quite confusing, honestly.

You can see the dependencies I used to get it working for both the (1) software.amazon.awssdk:* dependecies and the (2) com.amazonaws:* dependencies below:

dependencies {
    ...
    implementation 'software.amazon.awssdk:secretsmanager:2.21.0'
    implementation 'software.amazon.awssdk:sts:2.21.0' <------ (1) ---
    implementation 'software.amazon.awssdk:s3:2.21.0'
    implementation 'software.amazon.awssdk:dynamodb:2.21.0'
    ...
    implementation 'com.amazonaws:aws-java-sdk-sts:1.11.1004' <------ (2) ---
    implementation 'com.amazonaws.secretsmanager:aws-secretsmanager-jdbc:1.0.6'
}

[simonmarty]

You can't use the V1 version of the AWS SDK STS library ('com.amazonaws`) with version 2 of the JDBC caching library, and vice versa.

It's worth nothing that the Java SDK changed classpaths when moving from V1(com.amazonaws) to V2 (software.amazon.awssdk). However, additional libraries vended by AWS like the crypto SDK and this library did not change their classpaths. This was intentional, but I understand the confusion.

If you want to use the V2 SDK, make sure to add version 2 of this library to your project file, and vice versa.